DIFFERENCES OF THE SELBERG TRACE FORMULA AND SELBERG 
TYPE ZETA FUNCTIONS FOR HILBERT MODULAR SURFACES 



YASURO GON 

Abstract. We present the first example of the Selberg type zeta function for noncompact 
higher rank locally symmetric spaces. We study certain Selberg type zeta functions and 
bJQi Ruclle type zeta functions attached to the Hilbert modular group of a real quadratic field. 

We show that they have meromorphic extensions to the whole complex plane and satisfy 
functional equations. The method is based on considering the differences among several 
Selberg trace formulas with different weights for the Hilbert modular group. Besides as an 
application of the differences of the Selberg trace formula, we also obtain an asymptotic 
average of the class numbers of indefinite binary quadratic forms over the real quadratic 
integer ring. 
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1. Introduction 

In this article, we consider Selberg type zeta functions attached to the Hilbert modular 
group of a real quadratic field. First of all, we recall the original Selberg zeta function con- 
structed by Selberg in 1956. Let T be a co-finite discrete subgroup of PSL(2,R) acting on 
the upper half plane H. Take a hyperbolic element 7 G T, that is | tr(^y) | > 2, then the cen- 

/ Ni-i) 1 / 2 

tralizer of 7 in V is infinite cyclic and 7 is conjugate in PSL(2, K) to y ^ 

with A/" (7) > 1. Put Prim(r) be the set of T-conjugacy classes of the primitive hyperbolic 
elements in T. The Selberg zeta function for T is defined by the following Euler product: 

00 

Z r (s) := n( 1 ~ N(p)- {k+s A for Re(s) > 1. 

pSPrim(r) fe=0 

Selberg defined this zeta function and proved (Cf. Selberg [2UI22]) ■ 

(1) Z-p(s) defined for Re(s) > 1 extends meromorphically over the whole complex plane. 

(2) Z-p(s) has "non-trivial" zeros at s = ~ ±ir n of order equal to the multiplicity of the 

eigenvalue 1/4 + r 2 n of the Laplacian A = —y 2 (^2 + JpO acting on L 2 (r\H). 

(3) Zy{s) satisfies a functional equation between s and 1 — s. 

The theory of Selberg zeta functions for locally symmetric spaces of rank one is evolved 
by Gangolli [5] (compact case) and Gangolli- Warner [6] (noncompact case). For higher 
rank cases, Deitmar [1] defined and studied "generalized Selberg zeta functions" for com- 
pact higher rank locally symmetric spaces. (See also Kelmer-Sarnak [15J). Therefore, our 
concern is to define and study "Selberg type zeta functions" for noncompact higher rank 
locally symmetric spaces such as Hilbert modular surfaces. 

Let us explain our main results on Selberg type zeta functions for Hilbert modular 
surfaces in more detail. Let K/Q be a real quadratic field with class number one and 
Ok be the ring of integers of K. Put D be the discriminant of K and e > 1 be the 
fundamental unit of K. We denote the generator of Gel(K/Q) by a and put a 1 := o~(a) 

and N(a) := aa' for a G K. We also put Y = ( j ) for 7 = ( ° J) G PSL(2, O k ). 

Let = {(7,7') I 7 G PSL(2, Ok)} be the Hilbert modular group of K. It is known that 
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is a co-finite (non-cocompact) irreducible discrete subgroup of PSL(2,R) x PSL(2,R) 
and T^- acts on the product H 2 of two copies of the upper half plane EI by component- 
wise linear fractional transformation. Tk have only one cusp (oo, oo), i.e. IV-inequivalent 
parabolic fixed point. Xk '■= IV \H 2 is called the Hilbert modular surface. 

Let (7,7') G Tk be hyperbolic-elliptic, i.e, | tr('y) | > 2 and | tr('y') | < 2. Then the 
centralizer of hyperbolic-elliptic (7, 7') in Tk is infinite cyclic. 

Definition 1.1 (Selberg type zeta function for Tk with the weight (0,m)). For an even 
integer m > 2, we define 



00 -1 

e i{m ~ 2)w iV(p)- (fc+s) ) for Re(s) > 1 



(p,p') 



(p,p>) k=0 

Here, {p,p') run through the set of primitive hyperbolic-elliptic T^-conjugacy classes of 
IV, and (p,p') is conjugate in PSL(2,M) 2 to 

iV(p) 1 / 2 \ / cosw —sin a; 
N(p)~ 1/2 /' V sinu cos a; 

Here, N(p) > 1, 00 G (0, it) and u ^ 7rQ. The product is absolutely convergent for 
Re(s) > 1. 

Our main theorems on analytic properties of Zk{s; m) are followings. 

Theorem 1.2 (Theorems 15.31 and 16.51) . For an even integer m > 2, Z#(s;ra) a priori 
defined for Re(s) > 1 has a meromorphic extension over the whole complex plane. 

Our Selberg zeta functions Z^(s;m) have also "non-trivial" zeros or poles and they 
have connections with the eigenvalues of two Laplacians. Let Aq 1 ^ := —yf(-^j + J^y) and 

Am := — + J^r) + ^ m ?/2^ be the Laplacians of weight and m for (zi, z 2 ) G H 2 . 

Two Laplacians Aq 1 ^ and A„ act on L 2 is (Tx\IHI 2 ; (0, m)), the space of Hilbert Maass forms 
of weight (0,m). (See Definition 12.161 for definition). 

Theorem 1.3 (Theorem 15.31) . For an even integer m > 4, 

(1) Z^(s;m) has "non-trivial" zeros at s = ^±ipj(m) of order equal to the multiplicity 

of the eigenvalue | + pj(m) 2 of A^ acting on Ker(Am^), 

(2) Zk(s; m) has u non-trivial" poles at s = | ± ipj(m — 2) of order equal to the multi- 
plicity of the eigenvalue | + Pj(m — 2) 2 of A^ acting on Ker(A^_ 2 )- 

Here, 

Ker(A( 2 )) = {/ G L^(r K \B? ; (0, qj) \ A® f = |(l - |) / 

/or ? = m m( im-2 and A$ 2) : L 2 . s (r x \H 2 ; (0, q)) -> L 2 4S (r K \e 2 ; (0, g - 2)) zs a "weight 
down" Maass operator. For "trivial zeros" of Zxis^m), see Theorem \5.3[ 

On the contrary to the case of m > 4, Z#(s; 2) has no "non-trivial" poles. 
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Theorem 1.4 (Theorem 16.51) . Zk{s;2) has "non-trivial" zeros at 

(1) s — | ± ipj(2) of order equal to the multiplicity of the eigenvalue | + Pj(2) 2 of Aq 
ac*m<? on Ker(Af) = {/ e LLfjVW 2 ; (0,2)) | A< 2) / = o} ; 

(2) s = | ± i/ij(—2) of order equal to the multiplicity of the eigenvalue \ + /%•(— 2) 2 o/ 
A« acting on Ker^) = {/ e L 2 is (lV\H 2 ; (0, -2)) | A?> / = o} . 

Here, K^h L 2 dis (T K \W 2 ; (0,-2)) -»■ L 2 iS (r x \H 2 ; (0,0)) zs a "wei#M up" Maass operator. 
For "trivial zeros" of Zk{s;2), see Theorem \6.5\ 

Actually Z#(s;m) has infinite "non-trivial" zeros by the following "Weyl's law". 

Theorem 1.5 (Theorem 16. 13[) . For an even integer m > 2, let 

N+(T) := #{j | l/4 + Pj (m) 2 < T) 

for T > 0. Then we have 

^mf 7 )-!™- 1 ) T7T- 2 T (T-KX). 

Our Z/<(s;m) also satisfy a symmetric functional equation. 

Theorem 1.6 (Theorems 15.41 and 16. 6ft . The zeta function ZK{s;m) satisfies the functional 
equation 

Zk(s; m) = Zk(1 — s;m). 
Here the completed zeta function Z^s^m) is given by 

Z K (s; m) \= Z K (s; m) Z id (s) Z e]1 (s; m) Z par/sct (s; m) Z hyp2 / sct (s; m). 

Each local Selberg zeta functions corresponding to each T K -conjugacy classes of Y K are 
explicitly given. See Theorems \5.J\ and \6.6i for details. 

We also consider the Ruelle type zeta function. 

Definition 1.7 (Ruelle type zeta function for IV). For Re(s) > 1, the Ruelle type zeta 
function for Tk is defined by the following absolutely convergent Euler product: 

rk(s)-.= n^-^rT 1 - 

(p,p') 

Here, (p,p') run through the set of primitive hyperbolic-elliptic IV-conjugacy classes of 
F K , and (p,p') is conjugate in PSL(2,IR) 2 to 

, ,s // N(p) l/2 \ / cosw -sinw 
(P,P) ~ ^ N (p)-i/2 J' ^ sinw cosu 

Here, N(p) > 1, u e (0, 7r) and oj £ ttQ. 
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By the relation 

Rk(s) 

we have 



Z K (s;2) 
Z K (s + 1;2)' 



Theorem 1.8 (Theorem 16 .71) . The function Rk{s) has a meromorphic continuation to the 
whole C Rk{s) has a double pole at s = 1 and nonzero for Re(s) > 1. 

As a byproduct of Theorem II .6} we obtain a simple functional equation for Rk{s) and 
an explicit formula of the coefficient of the leading term of Rk{s) at s — 0. 

Theorem 1.9 (Corollary I6.10p . Let D be the discriminant of K and D > 13. Then, the 
function Rk{s) satisfy the functional equation 

R K ( S ) R K (- S ) 2 2 ^*) sin(7rs )2i?(X K )-2a 2 (r)-2a3(r) 

/ 7rs x2a 2 (r) f7TS , 2a 3 (r) f tU _ 1) £ ( 5 + T 

•sin — sin — 1 



V27 v 3 y v Us) 2 

and the absolute value of the coefficient of the leading term of Rk{s) at s = is given by 

{2tx)E{X k ) (2 5 log5) 2 



K-(o)l 



2 a 2 (r) 3 a 3 (r) ( £ 2 _ iy • 



Here, E(Xk) denotes the Euler characteristic of X^, e is the fundamental unit of K , 
( e (s) = (1 — e^ 23 )' 1 and a r (T) is the number of elliptic fixed points in for which 
corresponding points have isotropy groups of order r. For D = 5,8 or 12, See Theorem \6.8\ 
and Corollaru \6.11\ 

These analytic properties and functional equations of Z K (s; m) and Rr(s) are obtained 
by using the "differences" of the Selberg trace formula for Hilbert modular surfaces. The 
key point is considering the differences between two Selberg trace formulas with different 
weights. For this we shall extend the Selberg trace formula for Hilbert modular group V k 
with trivial weight (Cf. Efrat [2] and Zograf [26]) to that with non-trivial weights (Theorem 
I2.22p . Based on our Selberg trace formula for Tk with weight (0,m), we can treat and 
obtain the differences and double differences of the Selberg trace formula (Theorems 14.11 
and S3]). 

As an application of "Double differences of the Selberg trace formula" (Theorem 14. 4p . 
we obtain a prime geodesic type theorem (Theorem I6.14p and a generalization of Sarnak's 
theorem [2D] on class numbers of indefinite binary quadratic forms over Z to that for 
class numbers of indefinite binary quadratic forms over Ok- Put := {d G Ok \ 36 £ 
Ok s.t. d = b 2 (mod 4), d not a square in Ok, d > 0, d' < 0}. For each d G let 
hK{d) denote the number of inequivalent primitive binary quadratic forms over Ok of 
discriminant d, and let (xd, yd) £ Ok x Ok be the fundamental solution of the Pellian 
equation x 2 — dy 2 = 4. Put EK(d) := (xd + \/dyd)/2. 
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Theorem 1.10 (Theorem 16. 16p . For x > 2, we have 

h K (d) = 2Li(x 2 ) - Li(x 2 ^( 2 ))- Yl Li0 2Sj( " 2) ) 

deX>+_ 1/2<Sj(2)<1 1/2<Sj(-2)<1 

+ 0(x 3/2 /\ogx) (x->oo). 

Here, s,j{2){l — Sj{2) s j and Sj(— 2) ^1 — Sj(— 2) j are eigenvalues of the Laplacian A^ acting 

cm Ker(A 2 ) and Ker(i^l 2 ) respectively. See Theorem \l .4\ for definition o/Ker(A 2 ) and 
Ker(il"i 2 2 ) ). 

2. The Selberg trace formula for Hilbert modular surfaces with 

non-trivial weights 

2.1. Hilbert modular group of a real quadratic field. Let K/Q be a real quadratic 
field with class number one and Ok be the ring of integers of K . Put D be the discriminant 
of K and e > 1 be the fundamental unit of K. We denote the generator of Gal(lf/Q) by 

a and put a' := o~(a) and N(a) := aa' for a E K. We also put 7' = ^ ^ ^, j 

6 
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for 



Let G be PSL(2,M) 2 = (SL(2, M)/{±/}) 2 and H 2 be the direct product of two copies 
of the upper half plane H := {z G C | Im(z) > 0}. The group G acts on M 2 by 

^•2 = \9\, 92)-(z 1 , z 2 ) = — , — E M 

\cizi + di c 2 z 2 + d 2 J 

forg = ( 9u g 2 ) = ((2 ^ and z = (zi, z 2 ) G H 2 . 

A discrete subgroup r C G is called irreducible if it is not commensurable with any 
direct product I\ x T 2 of two discrete subgroups of PSL(2, R). We have classification of 
the elements of irreducible T. 

Proposition 2.1 (Classification of the elements). Let T be an irreducible discrete subgroup 
ofG. Then any element ofT is one of the followings. 

(1) 7 = (I, I) is the identity 

(2) 7 = (71,72) is hyperbolic |tr(7i)| > 2 and |tr(7 2 )| > 2 

(3) 7 = (71,72) is elliptic |tr(7i)| < 2 and |tr(7 2 )| < 2 

(4) 7 = (71,72) is hyperbolic- elliptic |tr(7i)| > 2 and |tr('y 2 ) j < 2 

(5) 7 = (71,72) is elliptic-hyperbolic |tr(7i)| < 2 and |tr('y 2 ) j > 2 

(6) 7 = (71,72) is parabolic <^> |tr(7i)| = |tr(7 2 )| = 2 



Note that there are no other types in T. (parabolic-elliptic etc.) (Cf. Shimizu [23] ) 
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Let us consider the Hilbert modular group of the real quadratic field K, 



r *= {(-hc sms .))IC b d )^M- 

It is known that IV is an irreducible discrete subgroup of G — PSL(2, M.) 2 with the only 
one cusp oo := (oo, oo), i.e. T^-inequivalent parabolic fixed point. X K = r^\IHI 2 is called 
the Hilbert modular surface. 

We can easily see that 

Lemma 2.2 (Stabilizer of the cusp oo = (00,00)). The stabilizer of 00 = (00,00) in Tk 
is given by 



«")'( £i))\«eO$,aeO K }. 



Definition 2.3 (Types of hyperbolic elements). For a hyperbolic element 7, we define that 

(1) 7 is type 1 hyperbolic whose all fixed points are not fixed by parabolic elements. 

(2) 7 is type 2 hyperbolic not type 1 hyperbolic. 

Lemma 2.4. Any type 2 hyperbolic elements oJTk are conjugate to an element of 

{7m=(q £ -k ) keN,aeO K } 
in V K . The centralizer of 7fc )Q in IV is an infinite cyclic group. 

Proof. See pp.91-93 in 0. □ 

By the above lemma, we may take a generator of the centralizer Zr x (7fc,a) as ^k ,p with 
fco6N and (3 G Ok- We also write k as /co(7fc,a)- 

Let Ri, R2, ■ ■ ■ , Rn be a complete system of representatives of the r^-conjugacy classes 
of primitive elliptic elements of IV. vi,v 2 , ■ ■ ■ ,vn (y £ N, f > 2) denote the orders of 
Ri, i? 2 , • • ■ , Rn. We may assume that Rj is conjugate in PSL(2, M) 2 to 

77- • 77- tnTT • tnTT 

cos — — sin — \ / cos — — sm — 



■ sm — cos — / V sin— cos — 



For even natural number m > 2 and / G {0, 1, • • • ,vj — 1}, we define the integers 
aii(m,j), ai(m,j) G {0, 1, • • • ,v s - 1} by 

m — 2 

1 + *i( — o — ) = a i( m d) (mod ^) 
(2-1) 2 

Z-tj-( — - — ) = a7(m,j) (mod Vj) 

We denote by r H i, T E , The, r EH and r H 2, type 1 hyperbolic Tx-conjugacy classes, 
elliptic Tx-conjugacy classes, hyperbolic-elliptic T^-conjugacy classes, elliptic-hyperbolic 
T^-conjugacy classes and type 2 hyperbolic Tx-conjugacy classes of Tk respectively. 
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2.2. Preliminaries for the Selberg trace formula. Fix the weight (m 1 ,m 2 ) G (2Z) 2 . 
Set the automorphic factor j-y(Zj) = f* z 3 . +d \ for 7 G PSL(2,R) (J = 1,2). 

+ J^r) + iwij C? = ^) ^ e ^ ne Laplariam of weight rrij for 



Let A 



U) 



-y 



2( d 2 



the variable Zj. 



Let us define the L 2 -space of automorphic forms of weight (m 1 ,m 2 ) with respect to the 
Hilbert modular group Fr. 

Definition 2.5 (L 2 -space of automorphic forms of weight (mi, 7712)). 
L 2 (F K \U 2 ; (m l7 m 2 )) := {/: H 2 C, C°° 

(i)/((7,y)(^i,^)) =^(«i) mi iy(^) m '/(«i.^) V (7,V) g r x 

3(A«, A( 2 )) G R 2 A« = A«/(^,^), AS /(^ 2 ) = A^M,^) 



iv\h 2 



f(z)f(z) d/j,(z) < 00. 



Here, d/i(z) = ^nd^ for z = z 2 ) G H 2 . 

We denote C^°(M 2 ) the space of compactly supported smooth functions on R 2 . 
Take $ G C^°(R 2 ) and introduce the point-pair invariant kernel k(z, w) of weight 
(mi,m 2 ) for $ (as (6.3) on [HI p.386]): 



(2.2) 



k(z, w) 



21 - wi 



22 - w 2 



2 1 



Im Im W\ Im z 2 Im w 2 

for (z,u?) = ((^1,^2), (w 1 ,w 2 )) G H 2 x H 2 . Here, 



Hi 



(m.i,m 2 ) 



(Z,1<;) 



%i,ma)( z i w ) := H mi (2i,t0i) H m2 (z 2 ,W 2 ) 



with 



for j = 1,2. The reason of the last equality is that mi,m 2 are even integers. (See [TO 
Definition 2.1, p.359] and [TU (5.1), p.349]). 



Definition 2.6. For $ G C 

(2.3) 

(2.4) 
(2.5) 



oo/Tn)2\ 



define 



$(t/7i + w 2 ,M; 2 + i; 2 ) JJ 
j'=i 



Q(w 1 ,w 2 ) : = 
(wi,w 2 > 0), 

g(ut, u 2 ) :=Q(e Ul + e~ Ul - 2, e U2 + e" 



V w i + 4 



Wo 



■U2 



2), 



.7 J 



rrij/2 



dv\dv 2 



h(n,r 2 ) :-- 



j( Ul ,u 2 )e i{riUl+r2U2) duxdu 7 



We can easily check that Q(wi,w 2 ) G C^°([0, oo) 2 ), g{u\,u 2 ) G C ( 



oo/Trp2> 



is an even 



function and h(r 1 ,r 2 ) G C 



oofin>2> 



is an even and rapidly decreasing function. 
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Proposition 2.7. 

(2.6) 

1\2 ff d 2 Q 

— {Xi -f 11, X 2 l 2 ) 

3=1 



(xi,x 2 > 0) 



7T/ 7 7 R 2 dwidw 2 



Xj + A + t + t 3 



dt\dt-2 



[ [ h( ri ,r 2 )e- l{riUl+r2U2) dndr 2 . 
J Jr 2 

Proof. See [TT1 p.386], |2, Proposition 2.2] and [261 (1-1.1)]. □ 



s/l 1 


y s 2 2 




cz x + d\ mi 


c'z 2 + d! 


mi 


c^i + d\ 2si 


c'z 2 + d'\ 2s2 (czi + d)™ 1 ( 


dz 2 + d') m2 



2.3. Eisenstein series. Let (mi, m 2 ) G (2Z) 2 , z = (21,22) = (^1 +%i,^2 + ^2/2) G H 2 
and (si,s 2 ) G C 2 with Re(s 1 ) , Re(s 2 ) > 0. We define, 

E( mi) m 2 )(z, si, s 2 ) := ^ 

Definition 2.8 (Family of Eisenstein series). For (mi,m 2 ) e (2Z) 2 , z = (zi,z 2 ) = (xi + 
Wi, x 2 + 23/2) G H 2 , s G C with Re(s) > 1 and fceZwe define 

(2.8) £ (mijma) (z,s;fc) := £ (mi , m2 ) («,s + 7^-—, s - — -J . 

Proposition 2.9. For Re(s) > 1, £/ie Eisenstein series E( mi . m2 )(z, s; k) is absolutely con- 
vergent and 

E{ mi , m2 ){iz,s; k) = j<y(zi) mi j 7 >(z 2 ) m * E( mi>m ^(z,s; k) 

for any 7 G Tk- Et mi>m2 \(z, s; k) is a common eigenfunction of A m \ and A^ 2 - 

Proof. See pp.38-44 in 0. □ 

Proposition 2.10 (Fourier expansion of Eisenstein series). Put 

L :— {I — (h, l 2 ) G K 2 I ha + l 2 a G Z Va G £> A -} 

and := ZiXi + Z 2 #2 /or 2; = (sci,^) G IR 2 . We write the Fourier coefficients as 

ai(y, s; k) for I G L: 

E {rnum2) (z, s;k) = J2 a i(v, s ; k ) e 2m{l ' x) . 
leL 

Then the constant term ao(y,s; k) is given by 

i_c iiih 1— 



„" ' — «J, 2 + ^ (mi>m2) ( S , fc) ^ 21 -^ 2 ^ 
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with 



m l+ m 2 r /<-, -, \ TV„ i TTik l\Tf„ i vrifc 



ink 



Here, the Hecke L-function L(s,x~k) is defined by L(s,X-k) '■= — l0gS |^( c )l s 

o^(c)co K C 

for k G Z. 

For / 7^ (0, 0), az(y, s; fc) gwen by 

. mi +mo , , _ , , _ i nik i , . _ nik -i 

(-l)^-*cri-2 8 ,- fe (0 7r 2s |/ i r + ^- 1 |/ 2 | s -^" 1 

VD L(2s, X - k ) T(s + ^ £ + Bgn(Zi)^) r(a - ^ + sgn(Z 2 )^) 
x W / sgn(il) .^i iS+ _^|__i(47r|/ 1 |i/ 1 ) W Bgn(Ja) .^L jS _ a ^__i(47r|Z2|y2). 
I/ere, W^^z) zs Whittaker's confluent hypergeometric function (see [251 Chapter 16] for 

definition), and 0"i_2 S -&(/) = ^ ^ ^ whereVV 1 is the inverse different of K . 

^— ' L/V(c)r s 

(See pi p.50]). 

Proof. The case of (mi, m 2 ) = (0, 0) is proved in [2]. For general case, we use the formulas 
(see [31 p.55] and 3.384 (9)]): 

, 1/2 r ( S -|) r ( S ) 



— = (-l) m / 2 7T 

i-oo |x + i| 2s - m (x + i) m v 7 



vm/2 1/2 1° 2 

j r( s + f)r( s -fv 



and 



e -2nipxy 7r s f I nl i/V -1 



|x + i| 2s " m (x + z) m r(s + sgn(p)* 2 , 

for 7^ p G R . The rest of the proof is quite the same as in pp. 47-50 in [2]. □ 

We can prove the following theorem and proposition by the similar method in pp. 58-64 
in 0. 

Theorem 2.11 (Functional equation). For any k G Z, Ef mi>m2 )(z, s; k) and ip(s, k) can be 
continued meromorphically to all of s G C. Moreover, we have 

E(rn ll m 2 )(z, 1 — S] —k) = </?( mi)jn2 )(l — S, —k) -E( mijm2 )(z, S] k), 

and 

<f(mi,m 2 )(s, k) <P( mi , m2 )(l ~ S, -k) = 1. 

Proposition 2.12. E^ mi)m2 ^(z,s;k) and <£>( mijm2 )(s, k) have no poles in Re(s) > \, except 
for finitely many in (|, 1] when k = 0. 
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For Y > 1 and m = (m 1 ,m 2 ) G (2Z) 2 , define 



(2.10) El(z, S] k): 



Ej^z.s^k) - a (y,s;k) iiyiy 2 >Y, 
E m (z, s; k) if y x y 2 < Y. 



For Re(s) > 1, we note that E^(z, s; k) is a square-integrable function on the fundamental 
domain for IV. 

Theorem 2.13 (Maass-Selberg relation). Let m,m/ G (2Z) 2 withm + mf = (0,0). For 
(s, k) 7^ (s', k') and (s, k) + (s', fc') ^ (1, 0), we /lave 

^ s+s '- 1 - Vm(s, k) <p m ,(s', k') Y~ s ~ s ' +1 



(2.11) =2VD\oge 



+ Sk.k' 



s + s' - 1 

tp m '(s' : k') Y s ~ s ' - (frn(s, k) Y s '~ s 



Proof. See pp. 66-69 in [2]. □ 

2.4. Selberg trace formula for Hilbert modular surfaces. We consider a certain 
integral operator /Cr acting on L 2 (IV\IHI 2 ; (m 1 ,m 2 )). The kernel of this integral operator 
is given as follows. 

Definition 2.14 (Automorphic kernel function). For (z, w) G H 2 xH 2 and m = (mi, m 2 ) G 
(2Z) 2 , define 

K T (z 7 w) := k ( z ^ w )h( w ) 

(2 ' 12) h(( W > A f ™i + d \ mi { c'w 2 + d> ^ 

(7,V)erv Vl 17 Vl 

Here, the point-pair invariant kernel k(z,w) is defined in ( 12.2ft . 
It is known that 

Proposition 2.15. Lei L^r^H 2 ; (mi, r^)) be the subspace of the discrete spectrum of 
/Cr and L 2 on (r^:\E[ 2 ; (mi,m 2 )) be the subspace of the continuous spectrum. Then, we have 
a direct sum decomposition of /Cr -invariant subspaces : 

L 2 (T K \M 2 ; (m u m 2 )) = Ll s (T K \m 2 ; ( mi ,m 2 )) ® L 2 con (T K \U 2 ; (mi,m 2 )) 
and there is an orthonormal basis {4>j}JL Q of L 2 dis {T _k-\H 2 ; (mi, 7712)). 
Definition 2.16 (Hilbert Maass forms of weight (mi,m 2 )). Let (m 1 ,m 2 ) G (2Z) 2 . We call 

L 2 Ais (T K \M 2 ; (mi, ma)) 
the space of Hilbert Maass forms for Tk of weight (mi,m 2 ). 
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To subtract continuous spectrum on L 2 on (r A '\IHI 2 ; (m 1 ,m 2 )), we introduce (see [21 p. 79] 
or |2S1 p. 1644]) 

Definition 2.17. For (z,w) G H 2 x H 2 and m = (m 1 ,m 2 ) G (2Z) 2 , define 
H r (z,w) : 



(2.13) 



* E f h 



nk Trk 
r + — , r — 



2 log e 2 log e 
x £? m ^, - + ir; k^j E_ m (w, - - ir; -kj dr. 



Let {0j}°^ o be an orthonormal basis of L^ is (T K \H. 2 ; (mi,m 2 )) and (\f\ \f ) G M 2 such 



that 



AS0,=A«0, and Ag^ = Af^. 



A (i) > |mi| 
J ~ 2 



mi 



Lemma 2.18. For any j, 

(2.14) 

Proo/. See PH (6.1), p.385]. 

Let us define the set of spectral parameters 

Spec(mi,m 2 ) := {(rf\ rf>)}™ 



which is a discrete subset of 

Imd — 11 llmi 



x RUi 



\m2\ — 1| ||^2| — 1| 



Here, we write = \ + {rf 1 ) 2 and rf 1 are defined by 
(2.15) 



Jf) ._ 
3 ' 



Af-| ifA-> 4 



(0 ^ 1 



j\-\f if Af < i, 



for I = 1,2. 

Theorem 2.19. Kr(z,w) — H-p(z,w) is a Hilbert- Schmidt integral kernel, that is 

\Kr(z, w) — Hp(z, w)\ 2 dfi(z) dfx(w) < 00. 



'(rv\n 2 ) 2 

Proof. Similar with the proof of Theorem 9.7 in [2] or p. 1644 in |26j . 
We shall assume that 



□ 



□ 



k(z,w)= / k {1 \z,v)k {2 \v,w)dfi(v) 

where, k^ and k^ are defined as (12. 2p . Then Ky — Hr defines a integral operator of trace 
class. So, we have 
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(2.16) 



i=o 



r K \H2 



where the left hand side is absolutely convergent. 

Our next task is to evaluate the right hand side of (12.161) explicitly. 
Hereafter, we assume that the test functions are written as follows. 

Assumption 2.21. We shall assume that the test functions are products of two separate 
functions that each involve only one independent variable. That is 



(2.17) 



h(r l ,r 2 ) = hi(ri) h 2 (r 2 ), g(u h u 2 ) = gi(u x ) g 2 (u 2 ), 

x 2 ) = $i(xi) ® 2 (x 2 ), Q(wi, w 2 ) = Qi(wi) Q 2 {w 2 ). 

Without loss of generality we may assume that $ x and $2 are real valued. 



Now we can state the Selberg trace formula for our cases. We give a proof of this theorem 
in the next section. 

Theorem 2.22 (Selberg trace formula for L 2 (Tx\HI 2 ; (0, m)) with m 6 2Z). Let g(ui, u 2 ) 
be an even function in c7?°(IR 2 ) and put h(ri,r 2 ) := g(u\, u 2 )e^ VlUl+r2U2 ^ du\du 2 , 

so that h is even, rapidly decreasing and analytic. 
Then we have, 



(2.18) 
Here, 
1(h) := 



h{rf\rf ) = 1(h) + II a (h) + II h (h) + 111(h). 

3=0 



voi(r^\H 2 



if 



g(ui,u 2 ) 



e 2 " 2 du\du 2 



+ E 



(7,7')er H : 



16vr 2 JJ R 2 sinh(tti/2) sinh(M 2 /2) 

vol(r 7 \G 7 )( ? (logiV(7),logiV(7 / )) 
(AT( 7 )i/2 _ jv( 7 )-i/2)(jv(y)i/2 _ AT( 7 ')-V2) 



E 



/?(6»i,6» 2 )er E 



+ E 



_ e -t0i+t(m-l)0a 

16i/ijsin^i sin #2 
logiV( 7o ) 



g(ui,u 2 )e "a + V^J 



^ e i(m-l)u) poo 



(7,^)er H E 



+ E 

(w',7')er B H 



AT( T )i/2 _ AT( T )-i/2 4 S in a; 



Lcoshtt,- — cos 20,- . 

m- 

^(logiV(7),M)e 2 



du\du 2 



logiV^) 



00 



g" _ g 2 ^ 



jV(y)l/2_ AT(Y)-l/2 4 s i na ;/ 



^(«,logiV(7'))e^ 



oc 



cosh it — cos 2a; 



cosh u — cos 2a;' 



du 



du, 
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II a (h):= VDA -4loge(\og2 + C E ) <?(0, 0) + loge / [g(u,0) + g(0,u)]du 
1 J Jo 

logs 



2ir 2 
+ 21oge 



r' r ; 

— (1 + in) + — (1 + ir 2 ) h(ri, r 2 ) dr 1 dr 2 



g{0,u) 



p—u/2 



m 

1 — cosh — u 



du 



ll h {h) := -41og£2_^ 2^ Hwlfe — Z^IYl c/(2fcloge,2Hoge) 



and 



|iV(e fc -e" fc )| 



+ 41oge^log(e fc -e fc ) #(2Hoge, 2/cloge) 



fc=i 



+ 21oge^ 

fe=l ^fcloge 
oo 

+ 21oge^ 

k=i J2kl °z e 



g(u, 2k\oge) + g(2k loge, tt) 



cosh(ii/2) 



sinh(w/2) + sinh(Hoge 



1 — cosh(m(u/2 — A; loge)) 
g(2k log e, w) r-r^ — — — ; du. 



-^(0,0)^(^0). 



irk 



sinh(w/2 — k loge 

nk \ <fi{o,m) , 1 



-,r 



2 loge 2 loge/ y? (0 , m ) 2 



( — Mr, dr 



The series and integrals converges absolutely. Here, A is the constant term of the 
Laurent expansion of (k( s ) at s = 1 and Ce is the Euler constant. The case of (0,m) = 
(0, 0) is proved by Zograf [26] and Efrat [2]. 



3. Proof of the Selberg trace formula for Hilbert modular surfaces 

3.1. Orbital integrals and the fundamental domain. In this section we prove Theo- 
rem E22 We recall Theorem E2S 



3=0 



KAz,z) - H r (z,z) 



d[i(z) 



Formally, we have 



K r (z,z)dfi(z) = / k(z,"/z)j y (z)diJ l (z). 

r *W 2 7 eConj(r x ) Jr-rV? 
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Here, we put T 7 be the centralizer of 7 in IV. To prove Theorem I2.22[ we calculate the 
orbital integral 



explicitly for each r^-conjugacy classes [7] of Y K . However, 7 is parabolic or type 2 
hyperbolic, the above orbital integral does not converge. 

Therefore, we introduce the truncated fundamental domain for IV. First we construct 
the fundamental domain iV of the group IV. (See Lemma \2. 21) . By direct calculation, we 
have, 

Lemma 3.1 (Fundamental domain of Too). Let D be the discriminant of the quadratic 
field K . We write (zi, z 2 ) = {x\ + iyi, x 2 + iy 2 ) E H 2 . 

(1) If D = l (mod 4), put 

Foo ■= {(x 1 +iy 1 ,x 2 + iy 2 )\0 < (l - j-) x 1 + (l + ^) x 2 < 2, < x x - x 2 < 2\f3, 
e~ 2 < 2/1/2/2 < e 2 }. 

(2) Otherwise, put 

Foo ■= {(xi + iyi,x 2 + iy 2 ) \ < x\ + x 2 < 2, < x\ - x 2 < 2vD, e~ 2 < y\/y 2 < e 2 }. 
Then iV is a fundamental domain for the group acting on H 2 . 
We define the standard truncated fundamental domain for T^- 

Definition 3.2 (Standard fundamental domain). Let Y > 1. 

(1) The fundamental domain F of T^, which is contained in iV, is called the standard 
fundamental domain for T^- 

(2) F Y := {(21,22) E F\yiy 2 < Y} is called the truncated standard fundamental 
domain for Tk- 

(3) Let 7 be a parabolic or type 2 hyperbolic element of IV. 



is called the truncated standard fundamental domain for the centralizer of 7 in Tk ■ 

3.2. Contribution of the identity, type 1 hyperbolic, elliptic and mixed elements. 

In this subsection, we compute the orbital integral 



explicitly, when 7 is the identity, an elliptic, a type 1 hyperbolic, a hyperbolic-elliptic, 
or an elliptic- hyperbolic element. We note that all the integrals are convergent for these 
elements. Let (mi,m 2 ) E (2Z) 2 . 




F, 



7 



U 



<5er 7 \r 




16 Y. GON 

• Identity term: By definition, we have 



(3.1) I{m 1 ,m 2 ) := / k(z, z) d^z) = / H( mum2 )(z, z) $(0, 0) cfyi(z) 

</iv\h 2 Jr K \w 2 

= (-i) mi+m2 voi(r^\e 2 )$(o,o). 

And $(0, 0) is given by (see p. 396 in 



$(0,0) 



7T Z 



d 2 Q 
dwidw 2 



v ^Tt?-ti 



mi/2 



V 4 + *i + *iJ Lv / 4Tt| + t 2 



V 4 + *i-*2 



m 2 /2 



4tt 2 J 7 r2 (e Ul / 2 - e -«i/ 2 )(e ti2 / 2 - e" M2 / 2 ) 



• Type 1 hyperbolic terms: For type 1 hyperbolic element (7,7') G IV, we denote it by 
7 for simplicity It is known that the centralizer of 7 in IV is a free abelian group of rank 
two. (See Theorem 5.7 in [21 p. 26]). We can easily compute (see also [21 p. 31]) 



(3.2) 



r 7 \H 2 



voi(r 7 \e 2 )^(iogiv(7),iogiv(y)) 

(JV( 7 )V2 - iv( 7 )-i/2)(iv(y)i/2 _ AT(y)-i/2 



• Elliptic terms: Let R G V k be an elliptic element. We may assume that R is conjugate 
in G to the element 



R(o 1 ,e 2 ) 



f ( cos 61 — sin 61 \ ( cos 6 2 — sin 6 2 
\ \ sin 61 cos 6*! / ' V sin 9 2 cos 9 2 



Let .Ro be a generator of the centralizer of R in Tk and denote the order of Rq by ur. 
Then i? is conjugate in G to the element 



cos(7t/Vr) — sin(7r/^) \ / cos(t7r/V/j) — sin(t7r/z/R 



sin(7r/z/ fi ) cos(7r/^) /' \ sin(tTr/is R ) cos(tir/u R ) 



[v, 3t) = 1. 
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We write R = Rq with (1 < k < u R — 1), and put (<x,-, f3j) 
Using the formulas at pp. 389-394 in [TT] (see also p. 1647 in 

(3.3) 

E(mi,m 2 ;R) : = k(z, Rz)j R (z) dfi(z) 

J<R >\M2 



cos 9j, sin 9j) for j 
), we have 
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1,2. 



k((z 1 ,z 2 ),(r(9 1 )z 1 ,r(e 2 )z 2 )) 



{f5\Z\ + «i) mi ((3 2 z 2 + a 2 ) m2 dx 1 dy 1 dx 2 dy 2 



7T 2 

4z/ R /3i/3 2 Jo Jo 
1 



\P x z x + «i| mi |/3 2 z 2 + a 2 | m 2 

3 imi arg [2a i +iy / tl+4^f ] im 2 arg[2a 2 +jA/i2+4/3|] _ ^(^1; ^2) 



v /t? + 4/3 1 2 v ^f+4^ 



16^/3iA 



i(mi-l)e 



'}{ 



i e 



i(m 2 — 1)6*2 



on[« 



g{u 1 ,u 2 ) I |e 2^ 



cosh — cos 29 j 



dtidts 



du\du 2 . 



• Hyperbolic-elliptic terms: Let 7 = (7,7') G G be a hyperbolic-elliptic element. The 
group T 7 is infinite cyclic and there exists a generator 7 = (70, To) such that 7 = 7o with 
k > 1. We may assume that (70, To) is conjugate in G to the element 



1/2 



iV( 7c ^ 1/2 





yo) 



cos w — sin Uq 
sin u; cos u; 



Here, N(j ) > 1, u; G (0, 7r) and u ^ 7rQ. Using the formulas at pp.389 - 394 in [TT] (see 
also p. 1647 in [26]), we have 

(3.4) 
HE (mi, m 2 ; 7) : = 



r 7 \H 2 

k((zi,z 2 ), (N(~f)zi,r(u)z 2 )) 

<70>\H 2 

io g iv(7 ) r 

7V( 7 )V2_iV(7)-i/2 2sina;y_ 00 

iV(7) 1 / 2 + jV( T )-V2 + I mi/2 
,AT( 7 )l/2 + JV( 7 )-l/2 _ Wl . 



(iV( 7 ) 



-l/2\m 



1 (z 2 sinw + cosu;) m2 dxidyi dx 2 dy 2 



|jy( 7 )-i/2|mi l^sin^, + coso;| m2 y\ 



$ 1 (iV(7) + iV(7)- : 
c/f 1 



«m 2 arg [2 cos u+i •y/ t 2 +4sir 



$ 2 (t 2 



xAI+4 



sin 2 a; 



: dU 



log iV(7o) 



AT( T )i/2 _ j\T( T )-i/2 4 s i na; 



(7(logiV(7),n 2 )e^" 2 



e «2 g2ia; 



cosh u 2 — cos 2u 



du". 



• Elliptic-hyperbolic terms: Let 7 = (7,7') G G be an elliptic-hyperbolic element. The 
group T 7 is infinite cyclic and there exists a generator 70 = (70, To) such that 7 = 7 with 
I > 1. We may assume that (to>To) * s conjugate in G to the element 



cos cu'n — sin Wn 



sinw 



COS 0J, 







iV(T^ 1/2 



10) 







iv(^)- 1 / 2 
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Here, N(-f' Q ) > 1, cu' G (0,7r) and u' Q ^ 7rQ. Then we have 
(3.5) 



EH(m 1 ,m 2 ; / y) := 
logiV( 7o ; 



r 7 \H 2 



le 



i(mi-l)w' roc 



AT(y)i/2 _ AT(y)-i/2 4 S i n w / 



7 

log 7V( 7 ')) e - 



-Hi 



coshni — cos 2k/ 



du\. 



Putting together with the all results in this subsection, we obtain the term 1(h) in 
Theorem [27221 

3.3. Parabolic contribution. Let Tp be the set of r^-conjugacy classes of parabolic 
elements in Tk- Let (mi, 7712) G (2Z) 2 and Y > 1. We consider the parabolic contribution 
to the trace formula with the truncation parameter Y: 



P Y (m 1 ,m 2 ) 



E 

7 er P 



FX 



H z ,l z )h( z ) dfi(z). 



Here, F 7 r = \J Ser \ r 5(F Y ) and F Y = {(z 1 , z 2 ) G F| lm(zi) lm(z 2 ) < Y} is the truncated 
standard fundamental domain for Fx, which is defined in Definition 13.21 
Then we have 

Proposition 3.3. For m G TL and Y > 1, we have 
p y (0,m) =21oge logF#(0,0) 

POO 

^ -41og£(log2 + C E ) g(0,0) + hge \ \g(u,0) + g(0,u)]du 
logs 



2vr 2 
+ 21oge 



r r 1 

Lp-(1 + « r i) + + «r 2 ) ft.(ri, r 2 ) drirfr 2 



3(0,1*) 



ou/2 



-u/2 



m 

1 — cosh — u 
2 . 



du + o(l) (Y->oo). 



i/ere, t/ie constant term of the Laurent expansion of (k{s), the Dedekind zeta function 
of K , at s = 1 and C# := lim n _ s . 00 (l + | + • • ■ + - — logn) is the Euler constant. 

Proof. We recall that the test function $ is written as <&(xi,x 2 ) = $i(xi) $2(^2) with real 
valued $1 and $2 by Assumption 12.211 By course of the same procedure at p. 1648 in [26] 
(note that Zograf's 2a/D in [26] is VD in our notation), we have 

P Y (m ll m 2 ) 

$(u 1; u 2 ) du\du 2 



4v D(A_i log Y + A ) Re 
+ aVdA^ Re 

+ 0(1) (y->oo). 



f /"OO /-OO 


r(2 + rai)- 


mi 


r(2 + w 2 )l 


I Jo Jo 


. 2 + mi| . 




. 2 + iu 2 \ . 



Jo 



log(MiM 2 ) 



(2 + wi) i ™i r (2 + mg) 
.|2 + uti|J L|2 + ra 2 |- 



$(u 2 , u 2 ) duidur 
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Here, A_i,A are the coefficients of the Laurent expansion of (k{s), the Dedekind zeta 
function of K, at s — 1. In particular, A_ 1 = ^ . Put 



P (m 1 ,m 2 ) 

:= 4a/D(v4„i logF + A ) Re 
r D(A_ 1 logY + A )g(0,0) 




o Jo 



(2 + ™i r (2 + m 2 ) 
|2 + mi|J L|2 + 2ti 2 | 



« 2 ) du\du 2 



Here, the last equality is derived from Definition 12.61 
For j = 1, 2, put 




</o 



(2 + i«i)] m i r(2 + ra 2 ) 



|2 + mi| J L |2 + ut 2 | 



m2 



$(«!, w 2 ) du\du 2 \- 



Pj{rrii, m 2 ) := Re< 
We note that 

P Y (m 1 ,m 2 ) = Po(m 1 ,m 2 ) + 8 loge|Pi(mi, m 2 ) + P 2 (mi,m 2 )| + o(l). 

We calculate the case of (mi,m 2 ) = (0,m). 

■(2 + m 2 )" 



p 2 (0,m) = Re 




log(n 2 ) 



o </o 

oo 



L|2 + m 2 | 



Re< / $iK)dui / log(u 2 ) 



1 



m^) du\du 2 
(2 + m 2 ) 



12 + iu 2 \ 



$ 2 (m 2 ) du 2 



^(0) --(log 2 + C E )g 2 (0) + -/i 2 (0) 



47T 



h 2 (r 2 ) — (1 + ir 2 ) dr 2 



+ 



02(1*2) 



3M2/2 P-U2/2 



m 

1 — cosh — u 2 
2 



We refer to pp.406 - 411 in [TT] for the last equality. Thus, we obtain 



P 2 (0, m) = -i(log 2 + C £ ) (7(0, 0) + l - 



g(0, u) du 



1 



16tt 2 



r 



h(ri,r 2 ) — (1 + ir 2 ) dr 2 



1 

+ 4 



9(0, «) 



1 — cosh — u 



du. 



g«/2 g— m/2 

Similarly, we obtain 

1 1 //°° 1 

^(0, m) = --(log 2 + C E ) </(0, 0) + g ^ g(u, 0) du - 

The proof is finished. 



r 

h{ri,r 2 ) —(1 + iri) dri. 



□ 
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3.4. Type 2 hyperbolic contribution. Let (mi,m 2 ) G (2Z,) 2 and Y > 1. We consider 
the type 2 hyperbolic contribution to the trace formula with the truncation parameter Y: 

oo „ 



fc=i 7fc,aer H 2 



Here, ^k,a = ( q £ -k ) with k G N, a G are representatives of type 2 hyperbolic 
conjugacy classes of T^, given in Lemma [2 .4[ 

S y := {( Zl ,z 2 ) G F 7fc Q | Im(^)Im(z 2 ) < Y, Im(r(^)) Im(r'(^ 2 )) < Y} 
and (r, r') is a element of such that 

(r, r') - e~ k ), a'/((e') k - (e')- k )) = (oo, oo). 

We can show that (see [26, p. 1650]) 



F Y 

7fc,, 



k(z,-f k>a z)j lk (z)dfi(z) +o(l) (Y -> oo). 



Put r/fc := e 2fc + £ 2fe — 2 and recall that fco = fco(7fc.a) was defined after Lemma [2.41 We 
can compute (see [2j pp.91-97] or [261 (4.5.1), p.1650]) 

k(z,j kta z) j 7 (z) dn(z) 

S Y 

k -2\oge 



\N(e k - e~ fc ) 



+ ry fc ,x 2 + ?7 fc ) 



x 



2 log Y - log iV(A) 2 + log(x 2 + r] k ) + log(x 2 + %) 



X 



fVVk + 4 + za; 1 \ m i/2/ v / % + 4 + u; 2 \»W2 



a/%T4 - ix 2 - 



fcp • 21oge 

|N(e fc -e- fc ) 



Here, we put 

i2o(mi,m 2 ) := 



■|i?o(mi, m 2 ) + Ri(m 1 , m 2 ) + R 2 (m 1 ,m 2 ) 
$(x 2 + Tfc, x 2 + %) [2 log y - log iV(A) 2 

2X 2 \ m 2/ 2 



x r + ^ ) mi/2 ( ^jj*±I + ix > ) m2/2 dXldX2 , 

R j (m 1 ,m 2 ) ■= $(x 2 + r) k , x\ + r] k ) log(x 2 + 77*.) 

V / r^T4 + ix 1 \™i/2/ v ^T4 + ix 2 \^/2 
^ + 4-faJ Un-4-te J <fal<fc " = 1 ' 2) 



and A G 0r- such that the ideal (A) = (a, e k — e k ) 
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Firstly we note that 

R (m 1 ,m 2 ) = [21ogF-logiV(A) 2 ]g( % ,r/ fc ) 
= 2 (2 log F - logA^(A))c/(2fcloge,2fcloge) 



by using Proposition 12.71 and the formula g(u±, u 2 ) = Q(e Ul + e Ul — 2, e U2 + e U2 — 2). 
Hereafter let us compute the case of (mi,m 2 ) = (0,m). 

Proposition 3.4. Let m E 2Z. We have 

Ri{0,m) = 2\og(e k - e- k ) g(2k\oge,2k\oge) 
(3.6) f°° , , , , cos1i(m/2) 

Proof. We recall that the test function $ is written as <3>(xi,x 2 ) = $1(^1) $2(^2) by As- 
sumption (22H Therefore, 



i2i(0,m)= / $i(x? + 7fc)log(z? + 7fo)d2;i / $ 2 (x 2 + % ) f ^li + — ) m/2 <fa 2 

= ^l(%) ■ Q2(%) = • 5'2(2/clog5). 

Here, 11(77) = Ii(%) is given by 

h(v) = / / \og(x 2 + rj) Q[(x 2 + rj + t 2 ) dtdx 

n Jo J-oo 

The inner integral is evaluated as 

r*=* iog(y-t 2 ) r/ 2 . 

/ = at — 2 log(y cos # + 77 sin 0) rfb 1 

J~Vy=v VV-V-t 2 Jo 

= 2 j ' \ogT]d6 + 2 j ' \og(l + V -^- cos 2 0) ^ 

= 2,log^ + 2.1og 1+ ^;" )/ '' +1 

= 2.1og(^±^). 
Here, we used the formula: (see [91 4.399]) 

log(l + a sin 2 x) <ir = y log(l + a cos 2 x) dx = tt logy J for a > — 1. 
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Thus, we have 



h(v) = -2^°°{log( v ^+ y/rj) - log2}g;(y) dy 

POO POO 

= 2 log 2 / g[ (u) du-2 log(e" /2 - e~ u/2 + e k - s~ k ) g[ (u) du 

J2kloge J2k\oge 

= -2 log 2 9l (2k log e) + 2 9l (2k log e) log(2(e fc - 



/x cos1i(m/2) ^ 

fcioge 1 sinh(u/2) + sinh(Hoge) 



cosh(w/2) 



21og(e* - £ -) 9l (2Hog £ ) + / 2Mos/ .(«) sinll(u/2) + sjnh(Hog£) 



du. 



The rest is clear. 



□ 



Proposition 3.5. Let m G 2Z. We /jai>e 



(3.7) 



i2 2 (0, m) = 2 log(£ fc - e~ k ) g(2k log e, 2k log e) 
1 — cosh(m(u/2 — /clogs)) 



2k logs 



cosh(-u/2) 



sinh(w/2) + sinh(/clog£) 



sinh(u/2 — kloge) 



g(2kloge, u) du. 



Proof. We recall that the test function $ is written as <&(x\,x 2 ) = $i(xi) $2(^2) with real 
valued $1 and $ 2 by Assumption 12.211 Therefore, 



$i(xl + r) k ) dxt I § 2 (xi + Vk)\og(xl + ri k )( ^ k + A ^^) 1 dx 2 

V?7fc + 4 - ix 2 ) 



# 2 (0,m) = 
= Qi(Vk) ■ him) = 9i(2k\ogs) ■ h(r]k)- 



Here, I 2 (r]) = hiVk) is given by 



h(v) = — Re 



7T 



noo 
Q 2 (xi + V + t 2 ) hg(xj + V ) 
-00 



x 



yJxl + ri + A + t 2 -t\ m/2 f\/rj~+4 + ix 

\]x\ +7] + 4: + t 2 +t 



m/2 



dtdx 2 



— Re 

7T 



00 / /V2/ 



y/rf+A — ix 
* log(y-e) f ^+A + i^fy-y-e 

—<Vy-v-e\ 



dt)Q' 2 (y)dy 
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by changing the variables y = x\ + 77 + t 2 and £ = t. Next changing the variable £ = 
y/y — £ sin ip, we have 



— Re 

7T 

--Re 

7T 



oo rn/2 



r) J-n/2 
oo ptt/2 



lOg ((?/ — ?7) COS 2 <yC + 7?) 



y/r] + 4 + 7^/7/ - 77 COS 



/ , . 2 \ / srrm w + * cos V 

' / log((y-r/)cos ^ + 77) — 

J.^/a V cosh?/; + smy? 



Vi/ + 4 + y/y-v sin V 9 



dip Q' 2 (y)dy 



with 
(3.8) 



cosh 7/7 



y±± 

y-e 



sinh7x> 



7/ + 4 

y^l' 



Let us consider the integral 

r-Tr/2 



J(y) := Re 
Then we see that 



lt o \( s i nn w + * cos V 7 

10g((j/ - 77) COS If + 7])[ — 

7r / 2 V cosh w + sin tp 



dip 



1 r°° 1 1 r°° 

h = — J(v)4Mdy=--[J(v)QM]~ + - J'(y)Q 2 (y)dy 

K J v 7T 1 IT J v 

= -QM logr)dip+- J'(y)Q2(y)dy 

K J-n/2 7T ^ 

1 f 00 

= 21og(£ fc -£" fe )^(2fclog£) + - / J'(y)Q 2 (y)dy. 

Let us consider the function f(z) defined by 

Log((e-C)( 2i f i ) 2 + 77) 



/(*) 



to evaluate the derivative of J(£). Here, Log(z) is the principal value logarithm whose 
imaginary part lies in (— 7r, 7r]. 

Let e, 5 > be two sufficiently small real numbers, and define the closed curve C in 
the complex plane, which is made up of two semi-circular arcs starting from tp — — | + 8 
to = I — 5 of the radii 1 and e, and besides they are joined along by the straight lines 

<p = ±{\-S). 

Considering the counterclockwise contour integral of f(z) along the curve C, by Cauchy's 
integral theorem, we have 
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(3.9) 
(3.10) 
(3.11) 
(3.12) 



w / 2 ( sinh w + % cos ip 

j 2 \ wan W + Sin If 



w 2 \ cosh; 



l°g((£ ~~ v) cos2 V 9 + V J idip 



+ 
+ 
+ 



16 / ie w - z 



^ Log(K _, )( i^)%,)^ 



ze — ee 



7T/2 



V^e w + ee^ 



Log I (£-77) 



ee 



e -l e -i<PN 2 



?7 ) idy? 



ie™ + 



Log((e-^)(^^) 2 +^)^ 







Put e := w 73- — \ jz^i then we see that e satisfies (£ — ?7)((e 1 — e )/2)' 



77 and 



(13. 10p and f 1 3 . 1 2 j) are written as follows 
(J3I0D = 



e / in \ m f —1 o 

6 r iog ((-v)( y ~ y 



+ 



e w + y 



eo / e w _ y 



. Uy 

77 — Z7T > 

J 2/ 



log (77- (£-77) 



and 



dsn 



+ 



1 v e +v 



^X'-tt-')^)")}* 

W/ ^) m {>o g («-,)(^) 2 -,) + -}f. 



While, (13. lip is evaluated as 

»vr/2 



tt/2 



l + 0(e) log(l/e 2 ) + log 



£-77 



2i(p + 0{e 2 ) idip. 



Take the real part of 



-1) x 



and we obtain 

AO + 77 



e w + yj y 



yj y 



log 



£-77 



o(elog(l 
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Therefore, we can rewrite the above formula as follows. 
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71 



e w - y 
e w + y 



dy ^ 

h 71 

y 



e w + y 
e w - y 



1 



dy 

y 



-Trlogf^— -) +27rloge 



o( £ log(i)). 



Letting e — > +0, we have an expression for J(£). Changing the variable y = e u in the 
integral, we have 



i 





' (e w + e~ u \ 


du — 7T / 




J UQ 


\e w - e~ u ) 



- 1 



dll 



+ 2tt log 

with mq := logep 1 . Then we obtain an explicit formula for the derivative of </(£)• 



71 



2vr 



dw du 



' dw du 

1 sr + af 



w _|_ p— no 



e w _|_ e -« 



3 ui p— no 



By noting that 



e w _|_ g-n 

<9u> 1 y/rf+A du 

= ~ 2(t - v) vz+t' 2(e - r/) ve' 

<9£ ^ £ - v 



D U C — U 



with £ = e u + e~ u - 2. 
We obtain 

dj(0 



7T 



2tt £ fc e"/ 2 - £- fc e- u/2 



7r e e 



k u/2 



e e 



fc -u/2 



t u/2 e -kya + ( e u/2 £ -kym 
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Then, we have 

J'(e u + e~ u - 2) • (e u - e~ u ) 

7r(e"/ 2 + e""/ 2 ) n(e k e u l 2 - e~ k e~ u ' 2 ) 

~ e u/2 _ e -u/2 _|_ £ k _ £ -k + £_ Tj 



sinh(w/2) + sinh(fc loge) e~ k e u / 2 - e k e~ u / 2 

7TCOSh(u/2) 71 



|l — cosh(m(u/2 — k loge)) |. 



sinh(u/2) + sinh(/cloge) sinh(u/2 — A; loge) 
Substituting the above equality into the following, the proof is completed. 

1 f°° 

I 2 = 2\og(e k -e~ k )g 2 (2k\oge) + - / J'(OQ 2 (0^ 

1 f°° 

= 2\og(e k -e- k )g 2 (2k\oge) + - / J'(e u + e~ u - 2) ^(u) (e u - e~ u ) du. 



" </2fclog£ 

Putting together with the results in this subsection, we obtain 
Proposition 3.6. Form £ 2Z and Y" > 1, we /itroe 

oo 

i?,f(0,m) = 41oge log 7^g(2fc log e, 2 A; log e) 

) log(iV(a 

|iV(e fc - e- k 



□ 



fc=i 

00 k 



,i k (~f k:a )\og(N(a,e k -e k )) 
-41oge}^ ^ g(2k\oge,2k\oge) 

fc=1 7fc, Q er H 2 

oo 

+ 4 log e V] log(e fe - e~ fc ) a(2& log e, 2A; log e) 

k=l 

cosh(w/2) 



^ /»oo 

21ogeJ^ / 2A; log e) + g(2k loge, w) 

fc=l *'2fcloge L 



du 



sinh(tt/2) + sinh(A;loge 

^ r°° , 1 — cosh (m(u/2 — Hog e)) 

+ 21oge£ / g(2kloge,u) 1 i S 

fc=1 J2fcio g e smn(w// — /ciogej 

+ o(i) (y-»-oo). 

Proof. By noting the fact (see J2J Proposition 3.3, p. 97] or [221 p. 1650]) 

£ fc ( 7 ) = |iV(e fc -e" fc )|, 

76r H2 ,A r (7)=e 2fe 

the rest is clear. □ 
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3.5. Contribution from Eisenstein series. Let m 6 22 and Y > 1. Define the contri- 
bution from the Eisenstein series with the truncation parameter Y by 

EI Y (0,m):= [ H T (z,z)dfi(z). 

J F Y 

By using the Maass-Selberg relation (Theorem 12.131 ). we obtain 
Proposition 3.7. For m G 2Z, we have 

EI Y {0,m) =21oge logF ^#(2Hoge,2Hog£) 

irk \ V(o,m) / 1 



fcez 
h\r-\ 



nk 



Aix 21oge' 21og£/ V9 (0 , m ) V2 

+ ^(0, 0) tp i0 , m ) (\, 0) + o(l) (Y — >■ oo). 
Proof. By definition of the kernel function Hr(z, z), we can check that 
H r (z, z) dfi(z) 



+ ir.k) dr 



F Y 



dfi(z) 



fcez ^ 



7rA; 



r — 



2 log e ' 2 log e 



9>n\fD\oge J f 
+ o(l) (y-»-oo). 
Next we use the following special case of Theorem 12.131 



E 



Y 

(0,m) I 



z, — h ir, fc 



y 

(0,m) I *i 



z,~ + ir, k 



2v / J Dloge 



21ogF 



d/i(z) 

^(0,m) /l 



+ 5, 



0,fc" 



<£(0,m) ^2 

^(o,m)(| -ir,0) Y 



ir, k 

lit 



^(o, m) (| + ^,o) y- 2ir 



2ir 



Finally we obtain the desired formula as in the proof of Proposition 1.1 in [21 p. 85]. □ 

3.6. Cancellation of the logY terms. Let us complete the proof of Theorem 12.221 Let 
m G 2Z. By Propositions 13.31 13.61 and I3.7[ the log Y terms are canceled out and we have 



lim 

y-»oo 



E 



7Gr P ur H2 " F f 



Hr(z, z) dfi(z) 



FY 



(3.13) 



lim ^ P Y (0, m) + Hi (0, m) - EI (0, m) + oil) 

Y— >oo 1 



P y (0,m) +H*(0,m) 



Y=l 



EI Y (0,m) 



Y=l 



Y=l 



--: P(0, m) + H 2 (0, m) + SC(Q, m) 



28 Y. GON 

We see that P(0, m), H 2 (0, m) and SC(0, m) are identified with II a (/i), and III(/i) in 

Theorem 12.221 respectively. The series and integrals appearing in these terms are absolutely 
convergent by the assumption on the test functions h in this theorem. Thus we complete 
the proof. 

4. Differences of the Selberg trace formula for Hilbert modular 

surfaces 

4.1. Differences of the Selberg trace formula. Let m E 2Z. We introduce Maass 
operators Am and K m , which play important roles in considering the "differences" of the 
Selberg trace formulas. We refer to [UJ Proposition 5.13, p. 381] and [HJ pp. 305-307] for 
basic properties of these Maass operators. 

Firstly we consider the following "weight down" Maass operator 

^ ■= W2^~ ~V2^- + ^-- L 2 Ais (T K \lf ; (0,m)) -> L 2 is (IV\H 2 ; (0,m - 2)). 
ox 2 ay 2 2 

Recall that 

Ker(A^) = {/ G L 2 is (lV\H 2 ; (0,m)) | A% f = y (l - y) /}, 

i.e. \W = f (1 - f )-eigenspace. 

Let {\ + pj(m) 2 }'jLo be the set of eigenvalues of Aq 1 ^ acting on Ker(Am ), then we have 
a direct sum decomposition into eigenspaces of the Laplacians 

(4.1) Ker(A^) = 0LL(rx\H 2 ; (i + p,(m) 2 , ^ (l - |)) , (0, m)) . 

i=o 

Secondly we consider the following "weight up" Maass operator 

K%_ 2 := iy 2 J- + yj, A + : L 2 is (IV\H 2 ; (0, m - 2)) L 2 is (r A -\H 2 ; (0, m)). 

Recall that 

Ker(i,i 2 l 2 ) = {/ 6 L 2 is (^\H 2 ; (0,m - 2)) | A$f = ^ (l - ^) /}, 

i.e. A (2) = s(l - y)-eigenspace. 

Let {i + /ij(m - 2) 2 }°1 be tt 
have a direct sum decomposition into eigenspaces of the Laplacian 

oo 1 

(4.2) Ker(*£. a ) = L 2 is (lV\H 2 ; (± + Mi (m - 2) 2 , | (l - ^) ) , (0, m - 2 

j=0 

By considering the two kernel spaces (I4.1I) and (I4.2p . we subtract the Selberg trace 
formula for L 2 (r A \EI 2 ; (0, m — 2)) from the one associated with L 2 (r A \EI 2 ; (0, m)). Then 
we obtain (we give a proof in the next subsection) 



Let {j + jjLj{m — 2) 2 }°2 =0 be the set of eigenvalues of Aq 1 -* acting on Ker(i^i 2 ), then we 
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Theorem 4.1 (Differences of STF for L 2 (IV\H 2 ; (0,m)) - L 2 (T K \U 2 ; (0,m - 2))). Lei 
m G 2Z. VFe /iat>e 

oo 

i(m— 1) ' 



E ^ (p,(m)) ^(^V 1 ) - E ^ (^-(m - 2)) h 2 

3=0 j=0 

rr2 



2 



(m - l)/t 2 (^^) ^j^^ - J ri/ii(r 1 )tanh(7rri)dri 

„— i9i+i(m— 1)02 f 



cosh Mi — cos 26i- 



^ lQgiV(70) n j e i(m-l)^ 

— sgn(m — 1) log£<7x(0) h 2 



■ i(m— 1) ' 



2 



oo 



2sgn(m - l)\ogeh 2 {^^) S ^g 1 {2k\oge) £ ~ k \™- l \ 

k=l 



4.2. Proof of the differences of the Selberg trace formula. We prove Theorem 14. II in 
this subsection. (Basic strategy is the same as the case of the trace formulas for PSL(2, R), 
see [HI pp.481-485]). 

• Spectral side: 

By (14.11) and (14.21) . the difference between the spectral sides of L 2 (rV\H 2 ; (0,m)) and 
L 2 (rV\H 2 ; (0, m - 2)) is given by 

oo oo 

E ^ (PA™)) m*^) - E *n - 2 )) H^)- 

3=0 j=0 

• Identity term: 

Put I(m) := 1(0, m) — 1(0, m — 2). Here, I (mi, m<i) is defined in ( 13. ID . Then, we have (see 
pp.396-397 in [11]) 



I(m) 



4tt 2 7ieJir (e' ill/2 - e - Ul / 2 )(e U2 / 2 - e -" 2 / 2 ) 1 J 

voi(r K \e 2 ) 



^7T 2 



4. 2 ^tJ^^i*™^™** 

voi(r A -\e 2 ) f riMn) tanh(7rri) rfri /^je-MW^x i™- 1 ) 



(m - 1 ) ~^^2~ ' ^2( ' (m 2 1} ) / ri/ii(ri) tanh(7rri) dri. 
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Elliptic terms: 



Let R be an elliptic element. Put E(m;R) := E(0,m;R) — E(0,m — 2;R). Here, 
E(mi,m2] R) is defined in (13. 3p . Then, we have 

E{m-R) = - , - — // g(u 1 ,u 2 )e—\e^^-e- 2ld2 e^^\ 



16vr sin 6i sin 9 2 



n 



L L cosh it j — cos 29 j 



du\du$ 



g—i6»i+i(m— 1)6*2 

SvRsmOi sin 6* 2 

g— i0i+j(m— 1)6*2 



0i(«i)e' 



/ 9iK)e-" /2 



-«l/2 



coshw! — cos 



8z/fjsin#i sin 62 
Hyperbolic-elliptic terms: 



gUl _ g2«6l 



diti / g 2 (u 2 )e 2 " 2 rf M2 



cosh u\ — cos 2^i 



Let 7 be a hyperbolic-elliptic element. Put HE(m; 7) := HE(0, m; 7) — HE(0, m — 2; 7). 
Here, HE(mi,m 2 ;^f) is defined in (13. 4p . Then, we obtain, 



HE(m; 7) 



logiV(7o) ie i(m " 1)w Z" 00 



jV( 7 )V2 - JV(7) ~V2 4sina; 



g(\ogN(-r),u) e^ u \ 1 - e^e - " 



coshw — cos2w- 



logiV(7o) ie^"* -1 *" 



AT( 7 )V2 _ JV( 7 )-i/a 2 si 



sinu 



g(log iV(7), u) e m 2 "cfot 



AT( 7 )i/2 _ jV( 7 )-V2 
Parabolic contribution: 



2 sin a; 



Put P(m) := P(0,m) — P(0,m — 2). Here, P(0,m) is defined in ( I3.13p . Then we have, 

roo poo 

(4.3) P{m) = - log e^i(0) / g 2 (u) e s ^ lu du - g 2 {u) e~^ u du 

{.Jo Jo 

• Type 2 hyperbolic contribution: 



Put H 2 (m) := -^2(0, m) — -^2(0, m — 2). Here, -£^(0, m) is defined in A3. 13|) . Then we have, 

OO /»00 

21oge^ / g(2k\oge,u) 

k=l J 2k log e 

cosh((m — 2)(u/2 — A; logs)) — cosh(m(w/2 — /cloge)) 



# 2 (m) 



du 



log £ ^ 

fe=l > / 2fclog£ 



sinh(u/2 — fcloge) 
^(2A;loge,n)|e- (m - 1){ti/2 - fcloge) - e (™-i)(«/2-fcio ge ) j du 
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Therefore, we have 

oo 

H^mj = 21oge^3i(2fcloge) e k( - m ~ l) 

(4.4) 
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k=l 



. . m — 1 



2fc log £ 



g— k(m— 1) 



/•OO 

/ # 2 (w)e ^ . 

</2fclog£ 

Finally, we calculate the scattering contribution to the differences of the trace formula 
for Hilbert modular surfaces. Put SC{m) := SC(0, m) — 5(7(0, m — 2). Here, SC(0, m) is 
defined in f lXT3|) . 



Proposition 4.2 (Scattering contribution). 

oo 

SC (m) = —2 sgn(m — 1) log e ^ #i (2/c log e) 



k=l 



-k\m-l\ 



|m-l| 

g 2 ('«)e 2 







(4.5) 



_fc|m— 1| 



2fc log e 



2fclog£ 



|m-l| 

92\u)e 2 n <i M 



m-ll 
J L 



— 2 sgn(m — 1) loge (?i(0) / g2(u) e 2 "rfu 
Proof. Firstly, we can easily check that 
^(o,m)0,0) 



r( s ) 2 r( s -i) 2 



Ck(2s - 1) 



D i-2s n 4s- 2 r(s + h) r( a - f ) r(| - s) 2 (1 - 2 S ) ' 

by using the functional equation of the Dedekind zeta function (k{s)- Thus we have 
<^(o,m)(|,0) = 1. 

Secondly, by the explicit formula for (p(o,m)(s, k), (see f)2.9p ) we see that 



<fi(0,m)(s,k)<f( ,m-2)(s,ky 



■nik m 



2 log £ 2 



Trik 1 m 



2 log £ 1 2 



So we have 



V(0,m) <P(0,m-2) / 



2-f + 4 + T + <(r-5&) 



21og£> 



21og£^ 



m — 1 



21og£'' T V 2 / 



Therefore, we have 



SC(m) 



(4.6) 



4tt ^ / ^ 



7rA; 7rA; 
r H , r 



m — 1 



21oge' 21ogey(r-^) 2 + (^i)2 



-Ef " 



ttA; 

r + ; , r 



m — 1 



logs' Vr 2 + (^) 2 



dr. 
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Thirdly, we use the Poisson summation formula to calculate ( I4.6P further. Let us determine 
the sequence {a^} such that 

^ hl ( r + ic^) = ^ ak exp ( 2mkr ' l ° g£ 

Then 



Jo 7^ V loge/ vr 



loge 



(27r)^i(2fcloge) = 2 log eyi (2fc logs). 



7T 

So (14. 6p is written as 

21oge ^ , _ , r° , , _ 2fc i OKeir m-1 



(4.7) SC(m) = 1 (2fcloge) / Wr) e 2fclog 

4?r 7^ i-oo 



2 i (md)2 



dr. 



Finally, let us evaluate the following integral 



and 



for fceZ and /c > 0. Recall that 

/■oo 

h 2 {r) = 2 I g 2 (u) cos(ru) du 
Jo 

and (see 3.723 (2)]) 

7o r 2 + (^) 2 |m-l| 
for m/1 (we assumed that m G 2Z). Then we obtain 

POO 

I _ I m-1 

Jo = sgn(m — 1) / 02 (m) e ~^~ u du. 
Jo 

While, we have 

1 r°° r°° m—1 
h =~ / 02 (u) ■ 2cos(rn) cos(r • 2/doge) — , m _ lN9 (ircfat 

Jo JO ^ + ( — ) 

r"00 /"OO 



1 r°° r°° m—1 
- / 92{u) cos(r(n + 2fcloge)) drdu 
71 Jo Jo r + I - 2~~ J 

1 /•» roo m—1 
+ -/ / 2 (m) cos(r(n - 2fcloge)) _ rfrrftt. 

^ Jo Jo r + I - 2~ / 
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Therefore, we have 

4=sgn(m-l) / g 2 (u)e-^ {u+2klog£) du + sgn(m- 1) / g 2 (u)e J ^\ u - 2kXoge \ du 
Jo Jo 
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k\m-l\ 



■ sgn(m — \)e 



+ sgn(m — \)e 



\m— 1| r 



-fc|m-l| 



g 2 (u) e~ 1 ^ ±u du + sgn(m - l)e k ^ m ~^ 
92{u)e 2 n dw 



m-l 

g 2 (u) e ~ u du 



2k log e 



for fceN. We complete the proof. 



□ 



We can now put together with, the parabolic contribution (I4.3|) . the type 2 hyperbolic 
contribution (I4.4p and the scattering contribution (I4.5p . then we obtain 

Proposition 4.3. 

(4.8) 



P(m) + H 2 (m) + SC(m) = — sgn(m — 1) log£ <?i(0) h 2 



■ i(m— 1) ' 



i(m— 1) - 
21 2 • 



fc=l 



- 2 sgn(m - 1) log e 9i( 2k lo S £ ) £~ fc|m ~ 1 ' h 
Proof. By fETBl and (IQjl . we see that 

g 2 (u)e J ^ u du- I g 2 (u) e~ 1 ~~^ ±u du 



and 



P(m) = — sgn(m — 1) log£<7i(0) 



H 2 {m) = 2 sgn(m — 



fc=i 



_fc|m— 1| 



171-11 

g 2 [u)e 2 u d M 



2fc log e 



-fc|m-l| 



m-lj 



g 2 (u)e 2 



2fc log £ 



Thus we have 



P(m) + # 2 (m) + 5C7(m) 
= — sgn(m — 1) log £51(0) 



u 



m-l| 



g 2 {u)e 2 u du+ g 2 (u)e 2 "rf-u 



2sgn(m — 1) log £ ^ gi (2k log £) 



fe=i 
00 



-k\m— 1| 



|m-l| 

g 2 {u)e 2 



+ £- fe|m - 11 / ^ 2 ( M )e^ u c/M + £- fc|m - 11 



2k log e 



2fc loge 



g 2 (u)e 2 



The rest is clear. 

By using the above proposition, we complete the proof of Theorem 14.11 



□ 
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4.3. Double differences of the Selberg trace formula. We wrote down the differences 
of the Selberg trace formulas for L 2 (Tk\M 2 ; (0, m)) and L 2 (Tk\I1 2 ; (0, m— 2)) in Theorem 
14.11 Let us denote the above differences formulas as L(m) — Lira — 2). Next we assume 
that fr^ C^V ) ^ ^ anc ^ ^ C^^ ) 7^ 0, then consider the "double differences": 

(L(m) - L(m - 2))h 2 {^^y 1 - (L(m - 2) - L{m - 4))/ i2 (^)~ 1 . 
Then we have, 

Theorem 4.4 (Double differences of STF for L 2 (T K \U 2 ; (0,m))). Let m G 2Z. We have 

oo oo oo oo 

3=0 j=0 j=0 j=0 

vo\(T K \m 2 



iTT 2 



re 



rh\{r) tanh(7rr) dr 

101 p i{m— 2)02 roo 



Avv sin 9\ 



gi{u)e- u ' 2 



e u _ g2i0i 



du 



-k\m— 3| 



cosh u — cos 26>! 

_ V lQgiV(7o) 9l (logiV(7))e^^ 

(7^)er H E v ' ; v ' ; 

— log e g\ (0) ^sgn(m — 1) — sgn(m — 3) j 

oo 

— 2 log e 9i (2k log e) ^sgn(m — 1) g-^l" 1 - 1 ! _ S gn(m — 3) e~ 

k=l 

Proof. By direct computation. □ 

5. Selberg type zeta functions for Hilbert modular surfaces 

5.1. Selberg type zeta functions. Let (7,7') G be hyperbolic-elliptic, i.e, | tr(^y) | > 2 
and |tr(7')| < 2. Then the centralizer of hyperbolic-elliptic (7,7') in is infinite cyclic. 

Definition 5.1 (Selberg type zeta function for with the weight (0, m)). Let m > 2 be 
an even integer. The Selberg type zeta function for with the weight (0, m) is defined 
by the following Euler product: 

00 

Z K (s;m) := Y[ Y[(l - e i{m ' 2)wo N{p)~ ik+s) y for Re(s) > 1. 
(p,p0 fc=o 

Here, (p,p') run through the set of primitive hyperbolic-elliptic r^-conjugacy classes of 
T^, and (p,p') is conjugate in PSL(2,IR) 2 to 

, ( ( N(pf/ 2 \ / cos w -sin w 

{P,P> ~ U Nip)- 1 / 2 r V sin w cos w 

where, N(p) > 1, wo G (0, 7r) and wq ^ vrQ. 
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Theorem I6.14[ which we prove in the next section by using Theorem I4.4[ ensures that 
the Euler product is absolutely convergent for Re(s) > 1. Therefore, Z#(s;m) represents 
a holomorphic function on the half plane Re(s) > 1. We remark that the exponent is — 1 
in the definition, which differs from the original one. 

For an even integer m < 2, we see that 



Z K {s\m) = Z K (s;A- m). 

Thus, it is sufficient to consider Zk{s; m) for an even integer m > 2 by the above relation. 

We show that Z#(s;m) has a meromorphic extension to the whole complex plane by 
using Theorem 14.41 (double differences of the Selberg trace formula). 

5.2. Test functions. Let us consider the logarithmic derivative of Zk(s; m). For Re(s) > 
1, 

— log Z K (s; m) = — > > log Who)- -, — — — - 

(p,p') k=0 v - r/ 
oo oo 

(5.1) =~J2 EE lo ^W N(p)- kl N(p)- ls e* m - 2 ^ 

(p,p') k=0 1=1 

(p,pl) 1=1 



Usually, we introduce a certain test function h(ri,r 2 ) to get a meromorphic extension of 
the logarithmic derivative of the Selberg type zeta functions. 

We can check that the Selberg trace formula (Theorem 12.221) holds for the test function 
h(ri,r 2 ) which satisfies the following condition (see [21 p. 105] or |26l p. 1651]): 

(1) h(±r 1 ,±r 2 ) = h(r l7 r 2 ), 

(2) h is analytic in the domain | Im(ri)| < | + 5, | Im(r2)| < ^"^"^ + ^ f° r some ^ > 0, 

(3) h(ri,r 2 ) = 0((1 + \ri\ 2 + |r 2 | 2 ) ) for some 5 > in this domain. 

(4) g 2 {u 2 ) e C C °°(R). 

We remark that the last condition assures the absolute convergence of the geometric side 
of Theorem 12.221 in particular that of Ilb(h). 

Let us consider the following test function: Firstly, we fix real numbers Pi,(3 2 > 2, 
ft ^ (3 2 . For s EC, Re(s) > 1, We set 

( , 2) u (r) ._ W ~ (f ~ I) 2 ) ((& 2 ~ (* ~ I) 2 ) _ 1 <*(«) , c 2 (s) 



(r 2 + (s-±) 2 )(r 2 + /3 2 )(r 2 + /3 2 ) r 2 + (s - ±) 2 r 2 + ft r 2 + # 
with 

f ,_ {s-\f-ft (s - \f - ft 

Cl{s) - ft -ft ' C2(s) " ^F - - 
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(See Parnovskii [17] for this type test functions). The Fourier transform of hi is given by 




hi(r)e~ tru dr 




Secondly, we take g 2 (u) G C£°(IR) such that its Fourier inverse transform h 2 (r) satisfies 
^«(m-i)<j _^ q anc j ^ i(m-3) j _^ q Then we can easily check that our test function 
h(r 1 ,r 2 ) := hi(r{) h 2 (r 2 ) satisfies the above sufficient condition for Theorem 12.221 

Thirdly, let us assume that m > 4 for simplicity. We will treat the case of m = 2 in the 
next subsection. Let m > 4 be an even integer. Then we have 



Here, K)J_ 2 , K)J_ A are the weight up Maass operators and their kernel are given by 



for q = m,m — 2. The fact ( 15. 31) is deduced from Lemma 12.181 and that the Hilbert 



respectively. Here, Am , A m _ 2 are the weight down Maass operators and their kernel are 
given by 



(5.3) 



(Ker(li 2 l 2 ), Ker(/4 2 1 4 )) = 



({0}, C) ifm = 4, 
({0}, {0}) ifm>6. 



Ker(< ) 2 ) = {/ 6 Z& (r^H 2 ; (0, q - 2)) | A« / = § (l - |) /} 




Ker(Af) = {/ 6 L^MH 2 ; (0, g)) | A«/ = |(l - |) /} 



for q = m, m — 2. If we set \j(q) :— \ + Pj(q) 



2 for q = m, m — 2, we note that 



(5.4) 



< X Q (q) < X 1 (q) < X 2 (q) < ... 



since Tk is irreducible and m > 4. 

Finally, we consider Theorem I4.4[ the double difference of the Selberg trace formula, for 
the above test function (]5.2p . Then we have 
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Theorem 5.2 (Double differences of STF for the above test function h± and h 2 ). Let 

m > 4 be an even integer. For Re(s) > 1, we have 



Ch[S, 



i 



1^2 +E „.., 



- L Pi (m - 2)2 + ( s - |)2 - ^ pi(m _ 2)2 + $ 



+ 5 



mA 



E 



fc=0 



\ + k 



J- Z' K (s\m) y-^ c h (s)Z' K (\ + p h ;m) 

2s - 1 Z K (s; m) 2f3 h Z K {\ + /3 h ; m) 



+ 1 ^y^^j - 1 - ai(m,j) - ai(m,j) ^fs + I 



2s 



+E^EE 



3=1 1=0 

N vj-1 



V 



3 



h=l j=l 1=0 



ai(m,j) - ai(m,j) (\ + Ph + l 

9 V 



V 3 



1 d , 



-(2s+m- 



' 4 ') 



E 



c fc (s) d f(l 



(1 _ £ -(2 s +m-2)) / ^ ^ 2/3 h d/3 fc bg \ (1 - £ -(2^+m-l)) J" 



_ -(2^+m- 



3)) 



i/ere ; f/H" 2 ) = T~l°gr(z) zs £/ie dig amma function. 

Proof. We compute each terms appearing in Theorem 14.41 for the test function h\ (r) given 
by dE2D. 

• Discrete spectrum: We denote the spectral side of Theorem 14.41 by A spec (s; m). By the 
fact (15 .3j) . we see that 



A spec (s;m) = y]fei(/Qj-(m)) - J^/ii(^(m- 2)) + 5 mi4 h^i/2). 

3=0 3=0 

• Identity term: We denote the identity contribution by Aid(s). By the proof of Propo- 
sition 4.9 in [ID], 



AM 



voi(rv\H 2 ) 



in' 



7rtan( ix[ s 

2 



^c /! ,(s)7rtan(vr/3 h ) 



h=l 



+E 



k=0 



s + k 



jri + E 



Ch(s) 



E 

h=l 



Ch(s) 



■kl 



oo 1 2 / 

=2&(-l) [rrr + E flTTX 
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Here, we used the partial fractional expansion of cot(7rz), the fact Ci(s) + 02(5) = —1 and 
the formula by Siegel (see Theorem (1.1) in [7] or Proposition 5.1 in [1]): 

voi(rv\H 2 ) 



(5.5) 



471-2 



2Cx(-l). 



Elliptic term: We denote the elliptic contribution by A e \\(s;m). This is given by 

i e -iOi e i(m-2)e 2 poo 



A,n(s: m) 



E 



K(0i,0 2 )er B 



E 



4z/r sin 9 1 

3 i(m— 2)02 poo 



vr Jo 



By noting (10.29) and (10.31) in p], we have 

AT Vj-lVj-l 



gx(u) cosh(w/2) 
cosh u — cos 20i 



cosh u — cos 20i 
du. 



du 



A P »(s:m) 



2.s 



^tEEE 

j=i k=i 1=0 



exp(i(m — 2){jiiktj)/vj) sin((2£ + 1) irk/vA fs + l 



c/ t (s) • < the same for s = \ + (3h >. 
h=i ^ > 



Next we use the following equality 



Uj-1 

E< 

fc=i 



-2)(mk h )/ Vj sin((2f + l)7Tfc/^-) 

j e i(2ai(m,j)+l)nk/uj j ie -i{2ai(m,j)+l)nk/u j 



2 (E an^fc/^ 



E 

fc=i 



sin(7r/c/z^) 



= z/j- - 1 - aj(j, m) - a7(j, m). 

Here, the integers ai(j,m), ai(j,m) E {0,1,... Uj — 1} are defined in (12. ip . The above 
equality is deduced from (see [3j p. 67]). 

u ~ l ie -i(2a+l)Trk/v 

} . , , , , =u-\-2a (a G {0,1,... z/j - 1}). 
Therefore, we have 



A e]1 (s;m) 



2s 



^ ^ i/j - 1 - ai(m,j) - ai(m,j) fs + l 



T EE^ 

i=i «=o 



Cfe(s) ^ ^ - 1 - a>i(m,j) - ati(m,j) /\ + (3 h + l 



2(3 h 

h=l j=l 1=0 
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• Hyperbolic-elliptic term: We denote the hyperbolic-elliptic contribution by v4 hyp _ eU (s). 
This is given by 

Ah n ( S -m)= — V lQgiV(70) N() -(s-l/2) i(m-2) U 



iV( 7 )i/2 _ jV( 7 )-i/s 

— Ch(s) ■ |the same for s = \ + (3^ 
1 Z' K (s; m) f c h {s) Z' K {\ + /3 h ; m) 



2s - 1 Z K (s; m) ^ 2f3 h Z K {\ + (3 h ; m) ' 

The last equality is derived from (15. ip . 

• Parabolic plus scattering term: Since m > 4, sgn(m — 1) — sgn(m. — 3) vanishes and 
this term contributes zero. So, A par / sct (s; m) = 0. 

• Type 2 hyperbolic plus scattering term: We denote the type 2 plus scattering contri- 
bution by A hyp2 / sc t(s;m). Then, 

A hyp2/sct (s;m) = _^g£^ £ -^-D (£ -^-i) _ £ -Hm-s )) 

k=l 



— c h (s) ■ < the same for s = | + (3 h > . 
h=i ^ ' 



Nothing that 



Z^ e £ - i _ £ -(2, +m -2) 2 logeds 8 V 



k=l 

for Re(s) > 1 — m/2. For Re(s) > 2 — m/2, therefore, we have 

2 



A hyp2/sct (a; m) = — — - log _ , 2s+m _ 2) , + ^ -=--nr log J 



2s - 1 ds 1(1- £ -(2 S +m-2)) j 2 p h d(3 h [(I - e -(2/9fc-Hn-l)) 

The proof is finished. □ 

5.3. Analytic continuation of Selberg type zeta functions. We prove 

Theorem 5.3. For an even integer m > 4, the Selberg zeta function Zj^{s\m), origi- 
nally defined for Re(s) > 1, has an analytic continuation to the whole complex plane as a 
meromorphic function. 

(1) Z^(s;m) has zeros at 

s — ~ ± ipj(m) of order equal to the multiplicity of the eigenvalue ~ + pj(m) 2 of 

Aq 1 ' 1 acting on Ker(Am), 

s = l- f + f^- o/ order 1 for k e Z. 
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(2) Zjc(s;m) has poles at 

s = ^±ipj(m — 2) of order equal to the multiplicity of the eigenvalue j+ pj(m — 2) 2 



of Aq 1 - 1 acting on Ker(A,^ 



m-2) ' 



n + t3z of order 1 for k G Z. 



2 1 log e 

(3) Zk{s; m) has zeros or poles (according to their orders are positive or negative) at 

^7V 



s = -k (k G N U {0}) of order (2k + l)E{X K ) + 2 EyLJV^i] " £/=i & 
(4) If m = 4, Z K (s,m) has additional simple zeros at s = and s = 1. 
Here, 

Ker(Af) = {/ G L 2 dis (rV\H 2 ; (0,g)) | A«/ = |(l - |) /} 

/or q = m or m — 2, and E(Xk) denotes the Euler characteristic of the Hilbert modular 
surface Xk and the definition of the integers /^(m) will be given in (15.6]) . W7ien £/ie 
location of two zeros or poles coincide, the orders of them are added. 

Proof. To get a meromorphic extension of Z^is; m), we show that the logarithmic deriva- 
tive of Zk{s; m) has a meromorphic extension to the whole complex plane and its poles are 
all simple with integral residues. By Theorem 15.21 it is easy to see that (2s — 1) A spec (s; m) 
and —(2s — 1) v4hyp2/sct(s; m) are meromorphic over the complex plane and their poles are 
all simple with integral residues. So, we consider the function: 



g(s; m) := -(2s - 1) (A id {s) + A ell (s; m)) 



We see that g(s;m) is also meromorphic and only have simple poles at s 
NU{0}. By the identity 

's + r 



-k for k G 



1=0 



V 



we have 
EM 



2s 



1=0 



4>(s) - log i/, 



ai(m,j) . (s + l 
V> 



- {^)-io g ^} + — J2 

3 1=0 



uj -2s - ai(m,j) - an(m,j) fs + l 



Thus, for k G N U {0}, (we write k = 1 + UjU with 1 = 0,1, 



, „ 3 



Res s= _ fe (2s - 1) Ej{s) 



2k 



ai(m,j) + aAm,j. 



21 



with I = k — Uj[k/uj]. Put 



(5.6) 



m 



on{m,j) + ai(m,j) - 21 



with I = k 



Pi 



ki 



Pi 



We see that (3kj(m) G Z since ai(m,j) + ai(m,j) = 21 (mod u 3 ) by (12. ip . 
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Therefore, we have 



N.N, N 

i r k 



Res s= . k g(s; m) = (2k + I)Ck(-I) - (2k + 1) ~ + J2^ 2 [~] + ^ ~ 

j=l 3 j—i 3 j—i 

N h N 

= (2k + 1) E(X K ) + 2 [-] - - &jM- 

3=1 3 3=1 

Here, E(Xk) is the Euler characteristic of the Hilbert modular surface Xk and we used 
the formula (see [7J Theorem (1.2), p. 60]): 



E(X K )=2( K (-1) + J2 



N 1 

Uj - 1 



V 



3 



3=1 

Hence the residues of g(s; m) are all integers. The rest of proof is clear. □ 
5.4. Functional equation of Selberg type zeta functions. 

Theorem 5.4. Letm > 4 be an even integer. The function Zk(s; m) satisfies the following 
functional equation 

Zk(s; m) = Zk(1 — s;m). 
Here the completed zeta function ZK(s,m) is given by 

Z K (s; m) := Z K (s; m) Z id (s) Z ea (s; m) Z hyp2 / sc t(s] m) 

with 

f \2Ck(-1) 

z-M) := (r 2 is r 2 (s + i 



N V4 — 1 , , .> , ■ \ 

J Vj—l—ai(m,j)—ai(m,j) 



3=1 1=0 



Z hyP 2/ sc t(s; m) := Cs(s + f - 1) Us + f - 2)~\ 

where, T 2 (z) is the double Gamma function (for definition, we refer to [2] or [HI Definition 
4.10, p. 751]), ui, u 2 , ■ • • i^n are the orders of the elliptic fixed points in Xk and the integers 
ai(m,j), ai(m,j) £ {0, 1, ■ ■ • , Vj — 1} was defined in ( \2.1\\ . ( e (s) := (1 — £~ 2s ) _1 and e is 
the fundamental unit of K . 

Proof. Starting from the formula in Theorem I5.2[ we compute the difference of the both 
sides at s and 1 — s. We see that 

. . . . , . / Z' K (s\m) Z' K (1 — s:m)\ 

2(k -1 ■ 2, - 1 tt cot (tts) + K ) + -P- '-J- 

\Z K (s;m) Z K {l-s)m)J 

+ ( Z' ell (s) + Z' cll (l-s) \ + f Z' hyp2/sct (s) + Z> hyp2/sct (l - s) 

\Z c n(s) Z e ii(l-s)J \Z hyp2/sct (s) Zhyp2/ S ct ( 1 - s) 



0. 
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by the partial fractional expansion: 7rcot(7rs) = YlkLo [j+k ~ i-s+fc ] • := ^Ca'( — 1; 

(2s — 1) 7rcot(7rs). It is known that the double sine function S2(z) := r 2 (2 — z) ^(z) - 
satisfies the differential equation (see [HJ Theorem 2.15, p.860]): 

— log Si(z) = —tt(z — 1) C0t(7Tz). 

Therefore, 

d , fatso, , / in rf i „/r 2 (2- s )r 2 (i- s ; 



^)-2C,(-l)^log(^ 2 ( S + l))=-2C,(-l)^log( 



+ - *) 



^id(s) Z id (l- s) 

Integrating and exponentiating, we obtain the desired functional equation. □ 

6. RUELLE TYPE ZETA FUNCTIONS AND APPLICATIONS 

6.1. Ruelle type zeta functions. We consider the following Ruelle type zeta function 

Definition 6.1 (Ruelle type zeta function for IV). For Re(s) > 1, the Ruelle type zeta 
function for Tk is defined by the following absolutely convergent Euler product: 

rk(s)-.= n^-^rT 1 - 

Here, (p,p') run through the set of primitive hyperbolic-elliptic T^-conjugacy classes of 
Tk, and (p,p') is conjugate in PSL(2,IR) 2 to 

, ff N(p) l/2 \ / cosw -sinw \\ 

(P,P)~{{ 7V( p )-i/2 )>{ sina; cosu ))■ 

Here, N(p) > 1, u e (0, n) and u £ ttQ. 

We note that the following relation between the Ruelle type zeta function and the Selberg 
type zeta function for IV. 

Lemma 6.2. For Re(s) > 1, we have 

Z K (s;2) 



Rk(s) 



Z K {s + l;2Y 
Proof. For Re(s) > 1, we have 

z K {s-,2) rw } nr= (i - ^(p)-^)- 1 



+ 1; 2) n (py) nr=o(! - A^)-( s+fc+i > 



□ 



To get a meromorphic extension of Rk{s), we consider meromorphic extension of the 
Selberg type zeta function Zk{s; 2). For this, we recall Theorem 14. A\ the double differences 
of the trace formula for the weight (0, 2). 
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Corollary 6.3 (Double differences of STF for L 2 (T K \U 2 ; (0, 2))). Let m = 2. We have 

oo oo 

X>(ft(2)) +j>i(A*i(-2)) -2/iiQ) 

i=0 j=0 

voi(r^\e 2 ) ro ° 



rhi{r) tanh(7rr) dr 



E 

R(9i,e 2 )er E 



-i6i 



Avr sin 



-u/2 



cosh u — cos 2#! 



dn 



(7,^)er HE 



fc=i 



idere, {l^ + p,,^) }j*L and {l/4 + /ij(— 2) }°5L are £ae sefo of eigenvalues of the Laplacian 



Aq acting on Ker(A;, ) and Ker(dCl 2 ) respectively. These kernel spaces are given by 
Ker(A< 2) ) = {/ e ^(r^H 2 ; (0, 2)) | A?>/ = 

Ker(^) = | / 6 LL(lV\H 2 ; (0, -2)) | A<3 / = o}. 



Proof. By noting that Ker(A[, 2 ^) = Ker(KQ Z> ) = C, the rest is clear by Theorem 14.41 □ 

Theorem 6.4 (Double differences of STF for the test function h\ (see (15.21) ) with the 
weight (0,2)). 

2 



'(2) 



E 

3=0 



E 



E 



1 



p,(2) 2 + ( S -i) 2 ^ Pj (2) 2 + /3 2 J ^L^.(-2) 2 + (,-i) 2 ^^.(-2) 2 + /3 

_i c h (s) 

U( s -l) + ^/3 2 -I 



2Cx(-l)E 



1 



E 



Ch(S 



1 Z^(s;2) , ' Ch (s)Z' K {\ + f3 h -2) 



2s 



* + ^ t^Ph + \ + k\ 2s-lZ K {s-2) f^2f3 h Z K {\ + (3 h -2) 

^EE 



k=0 
N vi-\ 



Va-1-21 [s + l 

V 



j=l 1=0 

d 



V 3 



+ ^M^*) + E 



+ 



2s - 1 ds 

1 d 
2s -Ids 



h=l 



Ch(s) d 
2(3 h d(3 h 



E°h{s) ^ Vj — 1 — 21 , fl + Ph + l 

h=l ' n j=l 1=0 3 



l0 g ( e -(2^+l)) 



log 



E 



c h (s) d 



log 



(l_ £ -2,)2j ^ 2/3, d/3, to ^(l- £ -(2^+l))2 
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Proof. By Corollary 16.31 and the same computation in Theorem 15.21 □ 

Theorem 6.5. The Selberg zeta function Zk{s;2), originally defined for Re(s) > 1, has 
an analytic continuation to the whole complex plane as a meromorphic function. 

(1) Zk{s;2) has a double pole at s = 1. 

(2) Zic(s;2) has zeros at 

s — | ±ipj(2) of order equal to the multiplicity of the eigenvalue | + Pj(2) 2 of A^ 

acting on Ker(A 2 2) ) = {/ G ^(M^ ; (0, 2)) | A< 2) / = o}, 

s = 2 =t i/ij(—2) of order equal to the multiplicity of the eigenvalue \ + fij(—2) 2 of 

A^ acting on Ker(K^) = {/ G ^(r^H 2 ; (0, -2)) | / = j. 

(3) ^(s; 2) /ias zeros at s = (A; G N) o/ order 2. 

(4) Z K (s; 2) /ias a zero at s = of order E(X K ). 

(5) Zk{s;2) has zeros or poles (according to their orders are positive or negative) at 
s = -k (k G N) of order (2k + l)E(X K ) + 2 E^it^M] ~ 2 kN. 

Here, E(Xk) denotes the Euler characteristic of the Hilbert modular surface X K . When 
the location of two zeros or poles coincide, the orders of them are added. 

Proof. By Theorem 16.41 and the same proof of Theorem 15.31 □ 

Theorem 6.6. The Selberg type zeta function Z K (s;m) satisfies the following functional 
equation 

Z K (s;2) = Z K (l-s;2). 
Here the completed zeta function Zx(s,m) is given by 

Z K (s; 2) := Z K (s; 2) Z id (s) Z cU (s; 2) Z par/sct (s; 2) Z hyp2 / sct (s; 2) 



with 



2Ck(-D N Vi ~ X 



Z*{s) := (r 2 (s)T 2 (s + 1)) K ~ , U;2):=nn r (f) ' ~* » 

j=l 1=0 

-^ P ar/sct( s ; 2) := e 2s , Z hyp2 / sct (s;2) := ( £ (s) 2 = (1 — e 2s ) 2 . 

Proof. By using Theorem I6.4[ the proof is the same as in Theorem 15.31 □ 

Theorem 6.7. The function Rr{s) has a meromorphic continuation to the whole C. Rk{s) 
has double pole at s = 1 and nonzero for Re(s) > 1. 

Proof. By Theorem 16.51 and Lemma 16.21 □ 
Theorem 6.8. The function Rk{s) has the following functional equation 

R K (s)R K (s)=(-l) EiX ^ (2^s)) mXK) fl(^^) 2 
(6.1) W V J 

'C^-i)C^ + i) N 



Us 



\2 
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where, ( e {s) = (1 — e~ 2s )~ 1 , N is the number of elliptic fixed points in Xk- 

Proof. By Theorem 16.61 we have 

Z K (s;2) Z K (-s;2) Z K (s;2) Z K (-s;2) 
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Rk(s)Rk(s) 



Z K (s + 1; 2) Z K {- 8 + 1; 2) Z K {\ - s; 2) Z K (1 + s; 2) 
£^(s)C^(s). 



The functions Bk(s) and Cx(s) are given as follows. 

Z id (l + s) Z id (l - s) Z cll (l + s; 2) Z cll (l - s; 2) 
K[S) ' Z id (s) Zid(-s) Z ell (s; 2) Z ell (-s; 2) ' 

/"» / \ ^par/set 

(1 + S; 2) Z p ar/sct(l — S; 2) ^hyp2/sct(l + 8] 2) ^hyp2/sct(l — 8] 2) 
■^par/sct(-5; 2) Z par / Sct ( — 8] 2) ^hyp2/sct (sj 2) ^hyp2/sct ( — s 'i 2) 

We can easily check that 



C K {s) 



C e (s-l)C e (s + l) 
Us? 



Let us compute Bk(s). Put 



and 



T 2 (s + l)T 2 (s + 2)T 2 (l-s)T 2 (2-s) 

r 2 (s)r 2 (s + i)r 2 (-s)r 2 (i- s ) 



V—\ j v -l-2l 

'S + t N 



Then we see that 

(6.2) B K (s) = E(s) E ^\ 



N 



j=0 



G Vl {l + 8)G Vi {l-s) "-0 

CIS) 3 



G„»G„,(-s) 

By using T 2 (s + 1)/T 2 (s) = v / 27rr(s)~ 1 (see [H] or [8j Proposition 4.11]), we have 

(6.3) S(s) = (27r) 2 (r(s) r(s + 1) r(-s) r(-s + l))" 1 = -4sin 2 (7rs). 

By using the multiplication formula for the Gamma function (see 0, 8.335]), we have 



g^i + s) c i ,(s)- 1 = r - 



v-1 

l i=i 



s + / 



V 



S + V 



V 



r i 



V 



V 



"(27r)^V/ 2 - s r( s ) 
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Hence, we have 



G v (l + s)G v (l-s) „. ._u=i 

-T— CIS » 



G v {s) G u ( 



(6.4) 



r i + - " r — 



V 



r(i + — 

V 



vY{s)Y{-s] 



(r(s)r(s + i)r(-s)r(-s + i))^ 

/ sin(7rs/^) 



rwr(i-sl 



T(s/v) r(l — u/s) \ sin(7rs) 
Substituting ( 16. 3D and ( 16. 4p into ( 16. 2p . we complete the proof. □ 

We can obtain an explicit formula of the leading term of Rk( s ) at s = 0. Let Uq denote 
an integer such that lim^o s~ U0 Rk (s) is a nonzero finite value and 

R* K (0) := \im S - n °R K (s). 

s— >0 

Theorem 6.9. The following equalities hold. 

n = E{X K ) + 2 

and 

K.(o)i=(2, ) ^>n,- i §^. 



i=i 



Proof. By Theorem 16. 8[ we can compute 



lim 



R K (s)R K (-s) 



E(X K )^2E(X K ) 



N 

n 



3=1 



(21oge) 



;i-^)(i- £ - 



□ 



, s 2(E(X x )+2) V 7 

The rest is clear. 

Corollary 6.10. Let D be the discriminant of K and D > 13. Then, the function Rr{s) 
satisfy the functional equation 

R K ( S ) Rk (_ 8 ) =(_1)^ W 2 2E ^ S i n ( vrs )2^(^)-2a 2 (r)-2a 3 (r) 
( 6 - 5 ) /7TS\2a 2 (r) /7rS N 2a 3 (r) l)^ £ ( s+ 1)\ 2 

■ s Kt) sin (T) ( — — J 

and the absolute value of the coefficient of the leading term of Rr(s) at s = is given by 

\r rmi _ ^) e{Xk) ( 2gl °g g ) 2 

1 * rWI 2 a 2( r )3 a 3( r ) (e 2 - l) 2 ' 

i/ere, a r (T) is the number of elliptic fixed points in Xk for which corresponding points have 
isotropy groups of order r. 

We remark that a 2 (T) and (T) are described by the class numbers of certain imaginary 
quadratic fields. (Cf. Prestel [IS] ) 
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Proof. By the fact that unless D = 5,8,12, then the a r (T) vanish for r > 3. (Cf. [TJ 
p.16]) □ 

Corollary 6.11. Let D = 5, 8 or 12. Then the class number of the real quadratic field K 
with the given discriminant D is one and the absolute value of the coefficient of the leading 
term of Rk{s) at s = is 

• (27r) 4 (2 2 3 2 5 2 )- 1 (2elog5) 2 /(£ 2 - l) 2 with e=(l + y/5)/2 ifD = 5, 

• (27r) 4 (2 2 3 2 4 2 )- 1 (2elog£) 2 /(£ 2 - l) 2 with e = 1 + V2 if D = 8, 

• (27r) 4 (2 3 3 2 6 1 )^ 1 (2eloge) 2 /(e 2 - l) 2 with 6 = 2 + ^3 if D = 12. 

Proof. Use the table of a r (T) for D = 5, 8 or 12 on [7, p.268]. □ 

6.2. Weyl's law. As an application of the double difference of the trace formula for V k 
with the weight (0,2) (Corollary 16.31) . we have the following "Weyl's law". 

Proposition 6.12 (Weyl's law I). Let T > 0. We consider the following counting function: 

N(T) : = #{j | 1/4 + Pj {2f <T} + #{j | 1/4 + N {-2f < T). 

Then we have 

/f> f>\ Ar(T , voi(rx\e 2 ) (T ^, 

(6.6) N{T) — T (T^oo). 

Proof. For any f3 > 0, the test function hi(r) = e _/3r2 is admissible in Corollary 16.31 The 
Fourier transform is 

9i(u) 



-u 2 /m 



so we have 

oo oo 
^ e -/3(l/4+p,(2) 2 ) + ^ e - /3 (l/4+ Atj (-2) 2 ) _ 2 

3=0 3=0 

voi(r^\e 2 ^ r°° 



8tt 



2 



e- /3(1/4+r2) rtanh(7rr)rfr 

oo 



e"^ 4 ie- ldl r ., a 



, u'/(40) e -u/2 



oo 



e" — e 



e-/3/4 ^ logiV(7o) p -(i og jv( 7 ))V(^) 



^(7) 1/2 - iV(7)" 1/2 
(7,oj)er H E 

e -/3/4 



cosh it — cos 2#i 



|21og£ +41og£ ^ e -(2fclog,) 2 /(4/3) £ -fe|. 



x/4^73 

Since tanh(7rr) = 1 + 0(e _27r ' r ') for any r 6 R, we obtain 

^ e -/3(l/4+p,(2) 2 ) + y^g-^(l/4+ W (-2) 2 ) = VQl(r^\H 2 ) _ 2lQg£ + Q ^ ^ _^ 

3=o j=o ^ ^ a/47t7J 
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By a classical Tauberian theorem, we complete the proof. □ 

We remark that the above proposition is enough to prove "a prime geodesic type the- 
orem" for the set of primitive hyperbolic-elliptic conjugacy classes of IV in the next sub- 
section. 

Besides we can prove a more strong Weyl's law by using Theorem I4.1[ the differences 
(not double differences) of the trace formula for IV with the weight (0, m). 

Theorem 6.13 (Weyl's law II). Let m G TL and T > 0. We consider the following two 
counting functions: 

N+(T) := #{j | 1/A + Pj (m) 2 < T), 
N'(T) :=#{j|l/4 + ^(m-2) 2 <T}. 

Then we have 

(6.7) If m> 2, then N+(T) ~ (m - 1) T (T->oo), 

(6.8) lfm<0,then N~(T) ~ (1 - m) ^£^S T (T -> oo). 

l07T 

Proof. We take the test function fai(r) = e _/3r2 (/3 > 0) in Theorem 14. 1L Then we obtain 
(by the same computation as in the proof of Proposition 16 . 1 2f) 



oo oo 
^ e -/3(l/4+ Pj (m) 2 ) _ 

3=0 j=0 



e -/3(l/4 +A1 ,(m-2) 2 ) 



The rest is clear. □ 

6.3. Prime geodesic theorem. We can show the following asymptotic formulas for 
counting functions of PIVe) the set of primitive hyperbolic-elliptic r^-conjugacy classes 
of T K , by Corollary 16. 3 1 and Proposition 16.121 

Theorem 6.14 (Prime geodesic theorem). For X > 1, 

(6.9) £ logiV( P H2*- Y, jm- £ f7=2T + °( Jf3/4 )- 

(p,p')6Pr HE l/2<a 3 -(2)<l jl ' \/2<8j(-2)<\ jl J 

N(p)<X 



(6.10) 



]T l=2Li(X)- ]T Li(X^ 2 ))- Yl Li(X s ^" 2 )) 

l/2<s 3 (2)< 

+ 0(X 3 / 4 /logX), 



(p,p')ePr H E 1/2<s 3 (2)<1 1/2< Sj -(-2)<1 

N(p)<X 
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where, Sj {2) := 1/2-^(2), s 3 {-2) := \-i^(-2) ; and U(x) : = /* 1/logt dt. {The 
condition 1/2 < Sj(2), Sj(—2) < 1 implies that Pj(2) 2 and fij(—2) 2 are negative. See 



Proof. We follow the same procedure as in Iwaniec [12]. Let us begin with the same test 
function on [121 P-155] or [131 P-401], given by 

gi(u) = 2 cosh(u/2) q(u), 

where, q(u) is even, smooth, supported on \u\ < log(X + Y), and such that < q(u) < 1 
and q(u) = 1 if \x\ < logX. The parameters X > Y > 1 will be chosen later. For 
s = 1 1 2 + %r in the segment 1/2 < s < 1 we have 

h 1 (r)= j (e su + e {1 - s)u \q(u)du = s~ l X s + 0(Y + X 1 / 2 ), 

J — oo 

and for s on the line Re(s) = 1/2 we get by partial integration that 

h^r) < Isl-^^minjl, |s|" 2 T 2 }, 

where T = XY^ 1 . By using Proposition I6.12[ the spectral side of Corollary 16.31 becomes 
(by the same method as in [T6| pp. 305-307]) 



oo oo 

j>i(pi(2)) +X>i(ft(-2)) -2/n(i/2) 



j=0 j=0 

1/2< Sj (2)<1 JV ' 1/2<Sj(-2)<1 JV y 

On the geometric side, the identity term contributes 

voi(r A -\e 2 



/oo 
rhi(r) tanh(Trr) dr < X 1/2 T 
-oo 



and the elliptic term, the parabolic plus scattering and the type 2 hyperbolic plus scattering 
terms contribute no more than the above bound. Gathering these estimates, we arrive at 

?(iogw(p)) 

(p,p')6Pr HE 

= -«+ E |(2)-+ E fp5)+^+^)- 

l/2<s J -(2)<l JV y 1/2< Sj (-2)<1 JV y 

Subtracting (16. lip from that for X + F in place of X, we deduce that 

log N{p) < F + X^T. 

x<Af( P )<x+y 

Hence, we can drop the excess over X(p) < X within the error term in (16.111) . As usual 
we choose Y = X 3 / 4 to minimize the error term. We completes the proof. □ 
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6.4. Binary quadratic forms over the ring of real quadratic integers. We denote 
by D the set of discriminants of integral binary quadratic forms, that is, 

V := {d £ Z | d = 0, 1 (mod 4), d not a square, d > 0}. 

For each d £ T>, let denote the number of inequivalent primitive binary quadratic 
forms of discriminant d, and let (xd, yd) be the fundamental solution of the Pellian equation 
x 2 — dy 2 = 4 over Z. Put 

_ x d + Vdyd 
e d :- • 

By using the prime geodesic theorem for PSL(2,Z), Sarnak [2U] deduced the following 
theorem on the average behavior of h(d). 

Theorem 6.15 (Sarnak [2"0l Theorem 2.1]). For x > 2, we have 

y j h(d)\oge d = h 0(x 3/2 (logx) 3 ) (x oo). 

dev 

e d <x 

Hd) = Li(x 2 ) + 0(x 3/2 (logx) 2 ) (x oo). 

d&V 

e d <x 

Here, Li(x) = 1/logt dt. 

There are several works to improve the error term of the prime geodesic theorem for 
PSL(2,Z). We refer to [24] for this subject. 

Let us consider a generalization of Theorem 16.151 to that for class numbers of indefinite 
binary quadratic forms over the real quadratic integer ring Ok- Put 

£> + _ : = {deO K \3be O k s.t. d = b 2 (mod 4), d not a square in Ok, d > 0, d' < 0}. 

For each d £ T> + _, let h,K{d) denote the number of inequivalent primitive binary quadratic 
forms of discriminant d over Ok, and let {xd,Vd) £ Ok x Ok be the fundamental solution 
of the Pellian equation x 2 — dy 2 = 4. Put 

x d + Vdy d 
£K{d) := • 

By using Theorem 16. 14^ we can deduce the following theorem on the average behavior of 
h K (d). 

Theorem 6.16. For x > 2, we have 

^ h K {d)\oge K {d)=x -- ^ ^ 

(6.12) d ^ V +- 1/2< Sj (2)<1 jK ' l/2< Sj (-2)<l Jl ' 

V ' e K (d)<x 

+ 0(x 3/2 ) (x->oo), 
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h K (d) =2Li(x 2 )- Yl Li(^ 3(2) )- E Li(x 2s ^) 

EK(d)<X 

+ 0(x 3/2 / log x) (x -)■ oo) 



(6 13) deV +- l/2< Sj (2)<l 1/2<Sj(-2)<1 



Proof. We recall the assumption that the class number of is one, so that Ok is a PID. 
Let Q(x,y) = ax 2 + 6xy + cy 2 be a primitive indefinite quadratic forms of discriminant 
d G over CV, i.e. a, 5, c G CV, the ideal (a, 6, c) = Ox and d = b 2 — 4ac > 0, d' < 0. 

The equation Q(6, 1) = has two real roots, Q\ = (—b + \fd)/2a and 62 = (— b — yd)/2a. 
By linear change of variable, SL(2, CV) acts on such forms and the number of equivalence 
classes is hxid). The stabilizer of Q under the action of SL(2, Ok) is equal to the stabilizer 
of Q\ or 6*2 and become a free abelian group of rank one. And a generator of this group is 
given by 

(t - bu )/2 -cu 



9 ^ \ au (t + bu )/2 

where, (t ,u ) is the fundamental solution to the Pellian equation t 2 — du 2 = 4 over Ok- 
Moreover the norm of g(Q) is SK{d) 2 with £x(d) — (to + u \/d)/2. 

The map Q 1— > (g,g') sends primitive O^-integral quadratic forms to hyperbolic-elliptic 
conjugacy classes of IV. It is known that it induces a bijection between classes of forms 
and primitive hyperbolic-elliptic conjugacy classes of IV. (We refer to Efrat [2] for details). 
Hence, we obtain the desired formula from Theorem 16.141 □ 
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